{
"cells": [
{
"cell_type": "markdown",
"id": "cff435f2-0261-4160-8e34-b5c56af2bb16",
"metadata": {},
"source": [
"# Evaluation Metrics for Classification\n",
"\n",
"* Use the same data as in the previous session: Churn Prediction (Identify clients that want to leave the company)\n",
"* Data https://www.kaggle.com/blastchar/telco-customer-churn"
]
},
{
"cell_type": "markdown",
"id": "a9c606de-f885-4881-ac03-2f91ee5dd3ff",
"metadata": {},
"source": [
"## Review of previous Session\n",
"\n",
"* Metrics: A function, that compares the actual values with the predicted values. It outputs a single umber that measures the goodness of the model\n",
"\n",
"### Setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d7a4bf0f-5824-4905-a374-b50d4279d4f1",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.metrics import mutual_info_score\n",
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"id": "d49cb1f5-a3e5-4f0f-877d-1d67d1284379",
"metadata": {},
"source": [
"### Load Data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9541f68c-d164-4097-a35e-c9894e4fba1d",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(\"../data/WA_Fn-UseC_-Telco-Customer-Churn.csv\")"
]
},
{
"cell_type": "markdown",
"id": "4b7472f5-def7-46c2-8d86-74f954f318c4",
"metadata": {},
"source": [
"### Data Preparation"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0715e0db-c953-47be-b645-97d7c5ab0570",
"metadata": {},
"outputs": [],
"source": [
"# Unify column names\n",
"df.columns = df.columns.str.lower().str.replace(\" \", \"_\")\n",
"# List of all categorical columns\n",
"categorical_columns = list(df.dtypes[df.dtypes == object].index)\n",
"# Unify categolrical columns\n",
"for c in categorical_columns:\n",
" df[c] = df[c].str.lower().str.replace(\" \", \"_\")\n",
"\n",
"# Change \"totalcharges to numeric and fill na with 0\n",
"df.totalcharges = pd.to_numeric(df[\"totalcharges\"], errors=\"coerce\")\n",
"df.totalcharges = df.totalcharges.fillna(0)\n",
"\n",
"# Change \"churn\" to type int\n",
"df[\"churn\"] = (df[\"churn\"] == \"yes\").astype(int)"
]
},
{
"cell_type": "markdown",
"id": "0e47b2cf-747a-4970-ba4d-f02cbfe79f8d",
"metadata": {},
"source": [
"### Model Setup"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "61ddad15-3767-4ac5-abff-b86e7bfa986c",
"metadata": {},
"outputs": [],
"source": [
"# 80% train + val = train_full, 20% test\n",
"df_train_full, df_test = train_test_split(df, test_size=0.2, random_state=1)\n",
"# 75% train, 25% val out of train_full \n",
"# 60% train, 20% val, 20% test out of df\n",
"df_train, df_val = train_test_split(df_train_full, test_size=0.25, random_state=1)\n",
"\n",
"# reset index\n",
"df_train = df_train.reset_index(drop=True)\n",
"df_val = df_val.reset_index(drop=True)\n",
"df_test = df_test.reset_index(drop=True)\n",
"\n",
"y_train = df_train[\"churn\"]\n",
"y_val = df_val[\"churn\"]\n",
"y_test = df_test[\"churn\"]\n",
"\n",
"# delete \"churn from df_train, df_val, df_test (not from df)\n",
"del df_train[\"churn\"]\n",
"del df_val[\"churn\"]\n",
"del df_test[\"churn\"]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "81d44f24-cb7a-4b52-8b83-cffdd07e395a",
"metadata": {},
"outputs": [],
"source": [
"numerical = [\"tenure\", \"monthlycharges\", \"totalcharges\"]\n",
"categorical = ['gender', 'seniorcitizen', 'partner', 'dependents',\n",
" 'phoneservice', 'multiplelines', 'internetservice',\n",
" 'onlinesecurity', 'onlinebackup', 'deviceprotection', 'techsupport',\n",
" 'streamingtv', 'streamingmovies', 'contract', 'paperlessbilling',\n",
" 'paymentmethod']"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "4ed56df1-a78f-4988-898d-6f82e0f2a0d0",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
"STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
"\n",
"Increase the number of iterations (max_iter) or scale the data as shown in:\n",
" https://scikit-learn.org/stable/modules/preprocessing.html\n",
"Please also refer to the documentation for alternative solver options:\n",
" https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
" n_iter_i = _check_optimize_result(\n"
]
},
{
"data": {
"text/plain": [
"LogisticRegression()"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# First convert to dictionary\n",
"train_dicts = df_train[categorical+numerical].to_dict(orient=\"records\")\n",
"\n",
"dv = DictVectorizer(sparse=False) # don't use sparse matrix\n",
"dv.fit(train_dicts)\n",
"X_train = dv.transform(train_dicts)\n",
"\n",
"val_dicts = df_val[categorical+numerical].to_dict(orient=\"records\")\n",
"X_val = dv.transform(val_dicts)\n",
"\n",
"model = LogisticRegression()\n",
"model.fit(X_train, y_train)"
]
},
{
"cell_type": "markdown",
"id": "a514ca21-d8b8-46c2-9590-1c6e5d3b9ec5",
"metadata": {},
"source": [
"### Predictions"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "f3963b4f-51dc-4c2b-8933-fa2ad8f7580d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8034066713981547"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred = model.predict_proba(X_val)[:,1]\n",
"churn_decision = (y_pred > 0.5)\n",
"(y_val == churn_decision).mean()"
]
},
{
"cell_type": "markdown",
"id": "c3aade31-a89c-4af7-a64a-3c6d39266d4f",
"metadata": {},
"source": [
"## Accuracy and Dummy Model\n",
"\n",
"* Evaluate the model on different thresholds\n",
"* Check the accuracy of the dummy baseline"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "548c0052-9abd-42ce-98fd-0088bad90367",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1409"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# How many validation values do we have?\n",
"len(y_val)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7bfe43fc-db2a-4f80-8373-4116ee4fdc3a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1132"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# How many correct decisions did we take?\n",
"(y_val == churn_decision).sum()"
]
},
{
"cell_type": "markdown",
"id": "03bc157d-37be-4159-ac1a-8073e082a340",
"metadata": {},
"source": [
"* Accruracy: Correct Prediction / All Predictions"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "94043d4b-ba0e-4964-9935-a9de58f26b1c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8034066713981547"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"1132/1409"
]
},
{
"cell_type": "markdown",
"id": "5dadb639-28c5-4a86-8c3b-3ff18a0d0aa1",
"metadata": {},
"source": [
"* Is the threshold of 0.5 good?\n",
"* Test the model with different thresholds"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "b0df11ad-abe4-4f59-b393-343b1889e597",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0. , 0.05, 0.1 , 0.15, 0.2 , 0.25, 0.3 , 0.35, 0.4 , 0.45, 0.5 ,\n",
" 0.55, 0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95, 1. ])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"thresholds = np.linspace(0,1,21)\n",
"thresholds"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "c8bc491a-bec8-43b7-b842-a0710395dc7d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"threshold: 0.00, score: 0.274\n",
"threshold: 0.05, score: 0.510\n",
"threshold: 0.10, score: 0.591\n",
"threshold: 0.15, score: 0.666\n",
"threshold: 0.20, score: 0.710\n",
"threshold: 0.25, score: 0.739\n",
"threshold: 0.30, score: 0.760\n",
"threshold: 0.35, score: 0.772\n",
"threshold: 0.40, score: 0.785\n",
"threshold: 0.45, score: 0.794\n",
"threshold: 0.50, score: 0.803\n",
"threshold: 0.55, score: 0.801\n",
"threshold: 0.60, score: 0.796\n",
"threshold: 0.65, score: 0.786\n",
"threshold: 0.70, score: 0.766\n",
"threshold: 0.75, score: 0.744\n",
"threshold: 0.80, score: 0.734\n",
"threshold: 0.85, score: 0.726\n",
"threshold: 0.90, score: 0.726\n",
"threshold: 0.95, score: 0.726\n",
"threshold: 1.00, score: 0.726\n"
]
}
],
"source": [
"scores = []\n",
"for t in thresholds:\n",
" churn_decision = (y_pred > t)\n",
" score = (y_val == churn_decision).mean()\n",
" print(f\"threshold: {t:.2f}, score: {score:.3f}\")\n",
" scores.append(score)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "b06d517f-5b3b-4914-9f96-3249960af188",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfV0lEQVR4nO3de3SddZ3v8fc3SZMmTZq2ubVN06T0SotAaSgCIhe5FB1F1qAic2Bk9CAqXmZ5Y85ZM7NmuWZGD47OOYDW6gDqUauCo/VMoSClgFShF6BQmqRp6CUtubRpm537ZX/PH3u3hJDS3XYnT/azP6+1srKf/fzY+/sju5/88nt+z/OYuyMiIqkvI+gCREQkORToIiIhoUAXEQkJBbqISEgo0EVEQiIrqDcuLi72qqqqoN5eRCQlbdmy5aC7l4y0L7BAr6qqYvPmzUG9vYhISjKzPSfapykXEZGQUKCLiISEAl1EJCQU6CIiIaFAFxEJCQW6iEhIJBToZrbCzGrNrN7M7h5hf6GZ/d7MXjaz7WZ2e/JLFRGRd3LSdehmlgncD1wDNAKbzGyNu782pNnngNfc/YNmVgLUmtnP3L1vVKoWGQVHu/t5ce9hth9oJzPDmJSTRX5OJvk5E5iUk0lB/Ht+Thb5E7PInZCJmQVdtshxiZxYtByod/cGADNbDdwADA10Bwos9unOB9qAgSTXKpI07s7rBzvZsucwW/ceZsuew+xs6eBUbg+QYTApOxbusfCPfZVNnkhVUR6zi/KoKppEZVEeU/KyR68zInGJBHo5sG/IdiNw0bA29wFrgANAAfAxd48mpUKRJOjuG2Rb4xG27D3M1j2H2br3CG2dsT8gJ0/M4oLKqXzw3Jksq5zKuRVTyDSjo3eAjt4BOnsHiPTEvnf2vfl46P7Y40EiPf08V3+QR7b2vOX9C3MnxEN+Uuz7tDyqimNhX5Kfo5G+JEUigT7SJ234OOY64CXgKmAu8ISZPevu7W95IbM7gDsAZs+efcrFiiTqjaPdbNkTG3lv3RObRhmIxj62c0sm8b5FpSyrnMqyyqnMLcknI+PtH/Pc7ExKCnJO6/17+gfZ29bF7oOdse+HOtlzqIuX9x3hv7YdIDrkX1Bediazp+VRWZTHwrICllZO5YKKqRTmTTit95b0lUigNwIVQ7ZnERuJD3U78E2P3c+u3sxeBxYBLwxt5O6rgFUA1dXVuvedJEX/YJQdb7S/JcAPHI2NkCdOyOD8iil8+vKzWFY5laUVU5k6afSnPyZOyGRBWQELygretq9vIMr+I93sPtTJ3kNvhv3Olg6eeK35eNjPL81nWeVULoj/4jmreJJG8vKOEgn0TcB8M5sD7AduBm4Z1mYv8D7gWTMrAxYCDcksVOSYts4+XozPe2/Zc5iXG4/Q0x+b4Sufksuyqmn899lTqK6cxqIZBUzIHF+rc7OzMphTPIk5xZPetq+jd4Bt+47E+rb3MGtfeYPVm2IznlPzJnDB7DcD/rxZU8jNzhzr8mUcO2mgu/uAmd0FrAMygQfcfbuZ3RnfvxL4BvCQmb1CbIrm6+5+cBTrljQRjTq7WjuOh/eWvYdpaO0EICvDWFJeyC3LK+Mj2SnMKMwNuOIzk5+TxSXzirlkXjHw1v4fO3j7ZE0LEOv/4pmTuWB2LOAvmjON0skTgyxfAmZ+Kof1k6i6utp1+VwZyd5DXTz+WhN/rD/I1j2Hae+JLZiaNin7eHgtq5zKubMKmTgh/Uaohzv7eHHfkL9Q9h2lu3+QzAzjpgtmcddV86iYlhd0mTJKzGyLu1ePuE+BLkFzd7YfaOfx7U08/lozNU0RIDaHXF017XiAVxXlaQ55BP2DUWreiPDI1kZ+/sJeolHnI9Wz+NyV85g1VcEeNgp0GXcGBqO8sLuNx7c388Rrzew/0k2GQXXVNK5dXMa1i6czu0hhdKqajvbw/Q31/OKFfTjOR6sr+NyV85g5JbWnouRNCnQZF7r7Bnm6rpXHX2tifU0LR7r6yc7K4L3zi7l2yXTet6iUovzTWyYob3XgSDff21DPLzftwzA+dmEFn71ybsofYxAFugSorbOPJ3c08/hrzTy7s5We/iiTJ2Zx9dllXLukjMvmlzApJ7A7IYbe/iPd3P9UPb/atI8MM265aDafuWIuZTp4mrIU6DJm3GOrMp7c0cKTNS1s3t1G1GFG4cTYVMqS6SyfM23cLSUMu31tXdz/VD0Pb2kkM+PNYC8tULCnGgW6jKregUGeb2hjfU0L62ta2NvWBcCi6QVcfXYZ1y2Zzjnlk3VAcxzYe6iLe9fv5Dcv7icrw7j13ZV8+vK5p31GrIw9BbokXUukhw01rTxZ08yzOw/S1TdITlYGl84r5qpFpVy5qJRyHYgbt3Yf7OTe9fX854uNZGdlcNvFVdx5+VymjcFZtHJmFOhyxqLR2NLCJ2uaWV/TwrbGo0BsKuWqRaVctaiUS+YW68zFFNPQ2sG96+v53Uv7ycvO4pPvmcOnLptDwURdR2a8UqDLaXF3NtS2sm57bFVKS6QXM1haMSUe4mWcPaNAUykhsLM5wneeqOPRV5uYkjeBz1w+l9surtIv6HFIgS6nxN15uq6Vf3u8jlf2H6UgJ4v3LijhqkWlXLGwREsLQ+yVxqN8+/Fanq5rpbQgh89fNY+PXTib7CwdxB4vFOiSsBdeb+Pb62p5YXcbs6bm8qWrF/Ch82bqH3Saeb7hEN9+vJZNuw8za2ouf3v1Aj68tJzMES4zLGNLgS4n9er+2MhsQ20rJQU5fEEjs7R37C+1e9bVsv1AO/NK8/nyNQtYcc50TbMFSIEuJ1TfEps7XftKbO70zsvn8teaO5UholHnse1N/Nvjtexq7eRd5YV85bqFvHd+sYI9AAp0eZt9bV387yd38putjeROyOSTl53Fpy6bw2StbpATGBiM8tuXDvDdJ+rYf6Sb5XOm8dXrFnJh1bSgS0srCnQ5rqW9h/uequcXL+zFzPjriyu58/K5OtApCesdGOSXm/Zx7/p6WiO9XLGwhI8sq+DiuUVaxz4GFOjC4c4+Vj6zix9v3M3AoPPRCyv4/FXzdLEmOW3dfYP8+E+7Wfn0Lo509QOweMZkLp1XxCXzilleNU3X6RkFCvQ01tM/yKpnGvjhMw109A3w4fPL+dLV86ksevvtz0ROR/9glG2NR9lYf5Dndh1k654j9A1Gycowls6ewiVzi7lkbhFLZ0/VQfYkUKCnqdqmCF9c/SI1TRGuXVzGl69dyMLpb79psUgydfcNsmXPYZ7bdZCN9Qd5Zf9Rog65EzK5cM40Lp1bxKXzilk8YzIZWgZ5yt4p0PX3UAi5Oz/50x7+Ze0OCiZm8eDtF3LlwtKgy5I0kZudyXvmF/Oe+bH7oh7t7ufPDYfYWH+QjbsO8a+P1gAwJW8CF59VxLLKqSyZWcjimZMpzNVB+TOhQA+Zgx29fO3hbayvaeHKhSX8r5vO05X0JFCFuRO4bsl0rlsyHYgdmN+46xDPxQP+0VebjretmJbLkhmFLJk5mSXlk1kys5DSghwtj0yQplxCZENtC1/59Tbae/r5n+8/m9surtQ/BBn3Dnb0sv1AO9sPHI1933+U3Ye6ju8vzs9m8cx4yM+MhXzltLy0na7RlEvI9fQP8s1Ha3ho424WlhXws09dpLlySRnF+TlcvqCEyxeUHH8u0tPPjjcib4b8gXZ++EwDA9HYADQ/J4uzZxRQMS0PI/WC/apFpXzg3BlJf10FeoobeuDzE5dUcff1i5g4QWd5SmormDiB5XOmsXzOmyct9Q4MsrO54y0h/3xDW4BVnr55pfmj8roK9BR17MDnP6/dwWQd+JQ0kJOVyTnlhZxTXhh0KeOWAj0FtUZ6+drDL/NUbStXLizhno+cR7HO9BRJewr0FPNUTQtfffhl2nsG+KcPLdGBTxE5ToGeIoYe+Fw0vYCfferdOvApIm+hQE8Brx/s5M6fbqG2OcLtl1bx9RU68Ckib6dAH+caD3dxyw//TO9AlIduv5ArdOBTRE4goSvlmNkKM6s1s3ozu3uE/V81s5fiX6+a2aCZ6SLJZ6gl0sN/+9HzdPYO8H8/eZHCXETe0UkD3cwygfuB64HFwMfNbPHQNu5+j7uf7+7nA38HPO3uqblAdJw40tXHbf/xAi2RXh68fTmLZ04OuiQRGecSGaEvB+rdvcHd+4DVwA3v0P7jwC+SUVy66ugd4BMPbqKhtZNVt1azrHJq0CWJSApIJNDLgX1Dthvjz72NmeUBK4BHTrD/DjPbbGabW1tbT7XWtNDTP8gdP9nMK/uPct8tS49fsU5E5GQSCfSRFjmf6IpeHwSeO9F0i7uvcvdqd68uKSkZqUla6x+MctfPt7Jx1yG+/ZFzuTZ+dToRkUQkEuiNQMWQ7VnAgRO0vRlNt5yWaNT5yq9f5g87WvjGDUu4cemsoEsSkRSTSKBvAuab2RwzyyYW2muGNzKzQuBy4HfJLTH83J2//92r/O6lA3xtxUJuvbgq6JJEJAWddB26uw+Y2V3AOiATeMDdt5vZnfH9K+NNbwQed/fOUas2hNydbz5Ww8+e38tnrpjLZ6+YF3RJIpKiEjqxyN3XAmuHPbdy2PZDwEPJKixdfG/DLn7wdAO3vruSr123MOhyRCSF6RbcAfrxxt3cs66WG5eW808fWqKLbInIGVGgB+SRLY3845rtXLO4jHtuOjdtb6clIsmjQA/AY6828dWHX+bSeUXc+/GlZGXqxyAiZ05JMsaeqWvlC794kfMrprDq1mpdNVFEkkaBPoY2727jjp9uZm5pPg9+YjmTcnSxSxFJHgX6GNl+4Ci3P7iJmYW5/ORvllOYNyHokkQkZBToY2BgMMqXVr9E/sQsfvqpiygp0P0/RST5FOhj4JGtjexs6eAf/mIx5VNygy5HREJKgT7KuvsG+c4TdSydPYUV5+hiWyIyehToo+zBja/T3N7L3SsW6cQhERlVCvRRdLizj+9v2MX7FpVy0VlFQZcjIiGnQB9F9z1VT2fvAF+/flHQpYhIGlCgj5J9bV389E97uGnZLBaUFQRdjoikAQX6KPnOE3WYwd9esyDoUkQkTSjQR8H2A0f57Uv7uf3SOcwo1DJFERkbCvRR8M1HayjMncBnrpgbdCkikkYU6En2x50HeXbnQe66ch6FuTq9X0TGjgI9iaJR55uP7aB8Si63XlwZdDkikmYU6En0+20HeHV/O1++dgE5WbosroiMLQV6kvQNRPn247WcPWMyHz6/POhyRCQNKdCT5GfP72FfWzd3X79It5MTkUAo0JMg0tPPvevruWRuEe+dXxx0OSKSphToSfCDpxto6+zj764/WxfgEpHAKNDPUEt7Dz/6YwMfPG8m75pVGHQ5IpLGFOhn6Lt/2Mlg1PnqtQuDLkVE0pwC/QzUt3Twq837+KuLKpldlBd0OSKS5hToZ+CedTXkTsjk81fNC7oUEREF+unasqeNddub+fR7z6IoXzd9FpHgJRToZrbCzGrNrN7M7j5BmyvM7CUz225mTye3zPHF3fnXtTWUFOTwycvmBF2OiAgAWSdrYGaZwP3ANUAjsMnM1rj7a0PaTAG+B6xw971mVjpK9Y4Lf9jRwuY9h/nnG88hL/uk/wtFRMZEIiP05UC9uze4ex+wGrhhWJtbgN+4+14Ad29Jbpnjx8BglG89VsNZJZP4WHVF0OWIiByXSKCXA/uGbDfGnxtqATDVzDaY2RYzu22kFzKzO8xss5ltbm1tPb2KA/bwlkbqWzr42nWLyMrUIQgRGT8SSaSRTn30YdtZwDLgA8B1wN+b2dvuvebuq9y92t2rS0pKTrnYoHX3DfLdP9RxwewpXLekLOhyRETeIpEJ4EZg6NzCLODACG0Ounsn0GlmzwDnAXVJqXKceOC512lu7+W+Wy7QKf4iMu4kMkLfBMw3szlmlg3cDKwZ1uZ3wGVmlmVmecBFwI7klhqso939rHx6F1efXcqFVdOCLkdE5G1OOkJ39wEzuwtYB2QCD7j7djO7M75/pbvvMLPHgG1AFPiRu786moWPtf/44+tEegb4sk7xF5FxKqE1d+6+Flg77LmVw7bvAe5JXmnjx9Gufh784+tcf850zp4xOehyRERGpGUaCfjRHxuI9A7wxavnB12KiMgJKdBP4khXHw8+t5sPvGsGi6ZrdC4i45cC/SR++GwDnX0anYvI+KdAfwdtnX08FB+dLygrCLocEZF3pEB/Bz98toGu/kG++D6NzkVk/FOgn8Chjl5+vHE3Hzx3JvM1OheRFKBAP4FVzzbQ3T/IFzQ6F5EUoUAfwcGOXn6ycQ8fOm8m80rzgy5HRCQhCvQRrHqmgd4Bjc5FJLUo0IdpjfTykz/t5obzy5lbotG5iKQOBfowP3h6F30DUd34WURSjgJ9iJb2Hn765z18eGk5Z2l0LiIpRoE+xPef3sVA1PnCVZo7F5HUo0CPa27v4WfP7+XGpeVUFU8KuhwRkVOmQI/7/oZdDEZdc+cikrIU6EDT0R5+/sJe/vKCciqLNDoXkdSkQAe+t6GeaNT5vObORSSFpX2gHzjSzeoX9nHTsllUTMsLuhwRkdOW9oH+vQ31RN353JWaOxeR1JbWgb7/SDe/3LSPj1RXaHQuIikvrQP9/qfqAbhLK1tEJATSNtAbD3fx6837+Gh1BeVTcoMuR0TkjKVtoN//VD2Gae5cREIjLQN9X1sXv97cyMcurGCmRuciEhJpGej3ra8nw4zPXjk36FJERJIm7QJ976EuHt7ayMeXVzCjUKNzEQmPtAv0e9fvJDPD+KzmzkUkZNIq0A919PKbF/dzy/LZlE2eGHQ5IiJJlVCgm9kKM6s1s3ozu3uE/VeY2VEzeyn+9Q/JL/XM7XgjwmDUuXZJWdCliIgkXdbJGphZJnA/cA3QCGwyszXu/tqwps+6+1+MQo1JU9PUDsDCsoKAKxERSb5ERujLgXp3b3D3PmA1cMPoljU66pojFOdnU5SfE3QpIiJJl0iglwP7hmw3xp8b7mIze9nMHjWzJUmpLslqmztYOF2jcxEJp0QC3UZ4zodtbwUq3f084F7gtyO+kNkdZrbZzDa3traeUqFnKhp1djZHWKDpFhEJqUQCvRGoGLI9CzgwtIG7t7t7R/zxWmCCmRUPfyF3X+Xu1e5eXVJScgZln7r9R7rp6hvU/LmIhFYigb4JmG9mc8wsG7gZWDO0gZlNNzOLP14ef91DyS72TNQ2RQBYoCkXEQmpk65ycfcBM7sLWAdkAg+4+3YzuzO+fyVwE/AZMxsAuoGb3X34tEygaptjgT6/ND/gSkRERsdJAx2OT6OsHfbcyiGP7wPuS25pyVXbFKF8Si4FEycEXYqIyKhImzNF65ojWuEiIqGWFoHePxhlV6uWLIpIuKVFoO8+2En/oGuFi4iEWloE+rEDolqDLiJhlh6B3hQhM8M4q2RS0KWIiIyatAn0qqI8Jk7IDLoUEZFRkxaBrhUuIpIOQh/o3X2D7Gnr0vy5iIRe6AO9vqUDd1ikEbqIhFzoA10rXEQkXYQ+0OuaI2RnZVBZpBUuIhJuoQ/0mqYI80vzycwY6bLuIiLhEfpAr2uK6AxREUkLoQ70o139NLX36BroIpIWQh3odS2xA6IaoYtIOgh1oB+7S5FOKhKRdBDqQK9rjlCQk8WMwolBlyIiMupCHeg1TREWTC8gfrtTEZFQC22guzt1zRGdUCQiaSO0gd4a6eVIVz8Ly3RTaBFJD6EN9OOn/OuAqIikifAGepOWLIpIegltoNc1RyjOz6EoPyfoUkRExkRoA722KcLC6Zo/F5H0EcpAj0aduuYOrXARkbQSykBvPNxNd/+g5s9FJK2EMtC1wkVE0lEoA71OdykSkTQUykCvbYowa2ou+TlZQZciIjJmEgp0M1thZrVmVm9md79DuwvNbNDMbkpeiaeurlk3tRCR9HPSQDezTOB+4HpgMfBxM1t8gnbfAtYlu8hT0T8YZVdrh+bPRSTtJDJCXw7Uu3uDu/cBq4EbRmj3eeARoCWJ9Z2y1w920j/oGqGLSNpJJNDLgX1Dthvjzx1nZuXAjcDKd3ohM7vDzDab2ebW1tZTrTUhx0751wFREUk3iQT6SBcT92Hb/w583d0H3+mF3H2Vu1e7e3VJSUmCJZ6auuYImRnGWSWTRuX1RUTGq0SWgTQCFUO2ZwEHhrWpBlbHbyRRDLzfzAbc/bfJKPJU1DZFqCrKY+KEzLF+axGRQCUS6JuA+WY2B9gP3AzcMrSBu8859tjMHgL+XxBhDrER+pKZhUG8tYhIoE465eLuA8BdxFav7AB+5e7bzexOM7tztAs8FV19A+xp69L8uYikpYTOvHH3tcDaYc+NeADU3T9x5mWdnvqWDtzRVRZFJC2F6kxRrXARkXQWqkCva46QnZVBZZFWuIhI+glVoNc2dzC/NJ/MjJFWWoqIhFuoAr2uSddwEZH0FZpAP9rVT1N7Dwt1DRcRSVOhCXTd1EJE0l3oAl1TLiKSrkIT6HVNEQpysphRODHoUkREAhGaQK9tjrBgegHx68mIiKSdUAS6u1PXHNEJRSKS1kIR6K2RXo509bOwTKf8i0j6CkWg18RP+V84fXLAlYiIBCcUgV53bMmiRugiksZCEei1TRGK83Moys8JuhQRkcCEItDrmiO6ZK6IpL2UD/Ro1Klr7tAKFxFJeykf6I2Hu+nuH9QZoiKS9lI+0Gua2gFdw0VEJOUD/c0VLgp0EUlvKR/otc0dzJqaS35OQrdHFREJrZQPdN3UQkQkJqUDvW8gyq7WDs2fi4iQ4oG++1AnA1HXCF1EhBQP9GPXcNEBURGRFA/0uqYImRnG3NJJQZciIhK4lA702uYIc4onkZOVGXQpIiKBS+lAr2vWChcRkWNSNtC7+gbY29al+XMRkbiEAt3MVphZrZnVm9ndI+y/wcy2mdlLZrbZzN6T/FLfqr6lA3d0lUURkbiTnl5pZpnA/cA1QCOwyczWuPtrQ5o9Caxxdzezc4FfAYtGo+BjarXCRUTkLRIZoS8H6t29wd37gNXADUMbuHuHu3t8cxLgjLLapgg5WRlUFmmFi4gIJBbo5cC+IduN8efewsxuNLMa4L+AvxnphczsjviUzObW1tbTqfe42uYI88vyycywM3odEZGwSCTQR0rMt43A3f0/3X0R8GHgGyO9kLuvcvdqd68uKSk5pUKHq2uOaLpFRGSIRAK9EagYsj0LOHCixu7+DDDXzIrPsLYTOtLVR3N7r5YsiogMkUigbwLmm9kcM8sGbgbWDG1gZvPMzOKPLwCygUPJLvaYuuYOQDe1EBEZ6qSrXNx9wMzuAtYBmcAD7r7dzO6M718J/CVwm5n1A93Ax4YcJE262vhNLTRCFxF5U0J3hXD3tcDaYc+tHPL4W8C3klvaidU2tVOQk8WMwolj9ZYiIuNeSp4pWtcUuwZ6fJZHRERIwUB3d2qbIyzU/LmIyFukXKC3RHo52t2v+XMRkWFSLtB1yr+IyMhSLtDzsjO5ZnGZplxERIZJaJXLeFJdNY3qqmlBlyEiMu6k3AhdRERGpkAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCRsFC9b/s5vbNYK7DnN/7wYOJjEclKB+pwe1Of0cCZ9rnT3Ee/hGVignwkz2+zu1UHXMZbU5/SgPqeH0eqzplxEREJCgS4iEhKpGuirgi4gAOpzelCf08Oo9Dkl59BFROTtUnWELiIiwyjQRURCYlwHupmtMLNaM6s3s7tH2G9m9n/i+7eZ2QVB1JlMCfT5r+J93WZmG83svCDqTKaT9XlIuwvNbNDMbhrL+kZDIn02syvM7CUz225mT491jcmWwGe70Mx+b2Yvx/t8exB1JouZPWBmLWb26gn2Jz+/3H1cfgGZwC7gLCAbeBlYPKzN+4FHAQPeDTwfdN1j0OdLgKnxx9enQ5+HtFsPrAVuCrruMfg5TwFeA2bHt0uDrnsM+vw/gG/FH5cAbUB20LWfQZ/fC1wAvHqC/UnPr/E8Ql8O1Lt7g7v3AauBG4a1uQH4icf8GZhiZjPGutAkOmmf3X2jux+Ob/4ZmDXGNSZbIj9ngM8DjwAtY1ncKEmkz7cAv3H3vQDunur9TqTPDhSYmQH5xAJ9YGzLTB53f4ZYH04k6fk1ngO9HNg3ZLsx/typtkklp9qfTxL7DZ/KTtpnMysHbgRWjmFdoymRn/MCYKqZbTCzLWZ225hVNzoS6fN9wNnAAeAV4IvuHh2b8gKR9PwazzeJthGeG77GMpE2qSTh/pjZlcQC/T2jWtHoS6TP/w583d0HY4O3lJdIn7OAZcD7gFzgT2b2Z3evG+3iRkkifb4OeAm4CpgLPGFmz7p7+yjXFpSk59d4DvRGoGLI9ixiv7lPtU0qSag/ZnYu8CPgenc/NEa1jZZE+lwNrI6HeTHwfjMbcPffjkmFyZfoZ/ugu3cCnWb2DHAekKqBnkifbwe+6bEJ5nozex1YBLwwNiWOuaTn13iectkEzDezOWaWDdwMrBnWZg1wW/xo8buBo+7+xlgXmkQn7bOZzQZ+A9yawqO1oU7aZ3ef4+5V7l4FPAx8NoXDHBL7bP8OuMzMsswsD7gI2DHGdSZTIn3eS+wvEsysDFgINIxplWMr6fk1bkfo7j5gZncB64gdIX/A3beb2Z3x/SuJrXh4P1APdBH7DZ+yEuzzPwBFwPfiI9YBT+Er1SXY51BJpM/uvsPMHgO2AVHgR+4+4vK3VJDgz/kbwENm9gqx6Yivu3vKXlbXzH4BXAEUm1kj8I/ABBi9/NKp/yIiITGep1xEROQUKNBFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQkFOgiIiHx/wEWiTpFN6hTxwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(thresholds, scores);"
]
},
{
"cell_type": "markdown",
"id": "b5863d4d-9faa-453e-84bf-bd07c341abe6",
"metadata": {},
"source": [
"* Here 0.5 is the best threshold, before and afterwards the accuracy is lower\n",
"* Scikit-learn provides a function for the accuracy (above we wrote our own function)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "e0b0c194-da7f-4800-9c80-c3f9dc01a040",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "15a4ffa1-c321-4cac-8454-938d466dc308",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8034066713981547"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"accuracy_score(y_val, (y_pred > 0.5))"
]
},
{
"cell_type": "markdown",
"id": "2eb06644-8976-4f7d-ba85-6371313bd1e7",
"metadata": {},
"source": [
"* Special cases: threshold = 0 and threshold = 1\n",
"* threshold = 1 means, we are predicting that no customer is churning\n",
" * For this case (Dummy Model) the accuracy is ~73%\n",
"* threshold = 0 means, we are predicting that all customers are churning\n",
"* Our model gives an accuracy of 0.8"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "5fcc8336-5ba7-49ca-a159-d20a1f6e1208",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.7260468417317246"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# non-churning users\n",
"1 - y_val.mean()"
]
},
{
"cell_type": "markdown",
"id": "55cd79f1-bbb4-4362-ba22-4310120c2489",
"metadata": {},
"source": [
"* In this data set that are much more non-churning than churning users\n",
"* We have class imbalance in this data set\n",
" * For class imbalance, predicting the majority class already gives quite good accuracy and accuracy can be a misleading metric"
]
},
{
"cell_type": "markdown",
"id": "739c4736-7a1e-4348-80c7-dbf753f256dd",
"metadata": {},
"source": [
"## Confusion Table\n",
"\n",
"* Different types of errors and correct decisions\n",
"* Arranging them in a table"
]
},
{
"cell_type": "markdown",
"id": "fd238f6d-a0c1-48c9-8e67-7696eb4c31bb",
"metadata": {},
"source": [
"![confusion_table](Screenshot1.png \"Confusion Table\")\n",
"![confusion_table](Screenshot2.png \"Confusion Table\")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "1f0809e8-5685-4e01-9562-ca5182b43cd0",
"metadata": {},
"outputs": [],
"source": [
"actual_positive = (y_val == 1)\n",
"actual_negative = (y_val == 0)\n",
"\n",
"t = 0.5\n",
"predict_positive = (y_pred >= t)\n",
"predict_negative = (y_pred < t)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "b5efff4b-f501-4517-ab8a-c5eb1c480349",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"210"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tp = (predict_positive & actual_positive).sum()\n",
"tp"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "76eb0d5e-477d-4707-a2da-9b8679729b15",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"922"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tn = (predict_negative & actual_negative).sum()\n",
"tn"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "c1280b4c-afcc-4974-8de7-7de00db78535",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"101"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fp = (predict_positive & actual_negative).sum()\n",
"fp"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "a58fcd48-35a5-4997-81d7-35381228ccd2",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"176"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fn = (predict_negative & actual_positive).sum()\n",
"fn"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "bfda861c-c0bb-47f9-bae5-af260995fc0d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[922, 101],\n",
" [176, 210]])"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"confusion_matrix = np.array([\n",
" [tn, fp],\n",
" [fn, tp]])\n",
"confusion_matrix"
]
},
{
"cell_type": "markdown",
"id": "af374736-2fdf-4618-b55d-6d7920b31a41",
"metadata": {},
"source": [
"* We have a lot of more false negatives than false positives\n",
" * False positives are predicted to churn, but they are not going to. We would send them a discount e-mail, though not necessary and would loose profit\n",
" * False positives are not receiving an e-mail and are leaving, also loosing profit in this case"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "7e9d4922-c2c6-4e0d-8499-71911cae226e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.65, 0.07],\n",
" [0.12, 0.15]])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# in relative numbers\n",
"(confusion_matrix / confusion_matrix.sum()).round(2)"
]
},
{
"cell_type": "markdown",
"id": "de4279d1-2c39-4f05-9a9b-42a10d698e6c",
"metadata": {},
"source": [
"![confusion_table](Screenshot3.png \"Confusion Table\")"
]
},
{
"cell_type": "markdown",
"id": "11ab231a-0bed-43f9-83e9-11c62ad8500f",
"metadata": {
"tags": []
},
"source": [
"## Precission and Recall\n",
"\n",
"* Very useful for binary clasification problems"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "45a945de-f6a2-4284-842a-6426369ec348",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8034066713981547"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Accuracy\n",
"(tp + tn) / (tp + tn + fp + fn)"
]
},
{
"cell_type": "markdown",
"id": "5d90ff83-3261-4dab-9a68-d5b686fd57e2",
"metadata": {},
"source": [
"Now we will look at other metrics:\n",
"* **Precision: Fraction of positive predictions that are correct**\n",
" * tp / (tp + fp)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "3dc1eb1b-a05c-4ab1-9205-2389a3444a28",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.6752411575562701"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"p = tp / (tp + fp)\n",
"p"
]
},
{
"cell_type": "markdown",
"id": "276bb7e7-d262-45c1-8d7c-e40e803df077",
"metadata": {},
"source": [
"* This means that (1-p)~33% will get a promotional e-mail, although they are not going to churn "
]
},
{
"cell_type": "markdown",
"id": "918b9191-7a9d-45f0-a9c4-db000310cfa4",
"metadata": {},
"source": [
"* **Recall: Fraction of correctly identified positive examples**\n",
" * tp / (tp + fn) = tp / (#positive observations)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "e454a016-f544-4022-9ad2-27c4ffe25021",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5440414507772021"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = tp / (tp + fn)\n",
"r"
]
},
{
"cell_type": "markdown",
"id": "8fd9970c-194f-43cc-a1b1-9b5e449ec833",
"metadata": {},
"source": [
"* This means that we failed to identify (1-r)~46% of people who are churning "
]
},
{
"cell_type": "markdown",
"id": "37e197a4-8ac7-40fe-a272-a9898aae2e0b",
"metadata": {},
"source": [
"## ROC Curves\n",
"\n",
"### TPR and FPR\n",
"\n",
"* TPR: True positive rate TP / (FN + TP) (Nr of true positives by all positives) == RECALL\n",
"* FPR: False positive rate FP / (TN + FP) (Nr of false positives by all negatives)"
]
},
{
"cell_type": "markdown",
"id": "62f61b68-5658-4884-9f80-9c21dacb6abf",
"metadata": {},
"source": [
"![confusion_table](Screenshot4.png \"Confusion Table\")"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "a45ed148-67e0-4669-9037-a5932df0d3d1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5440414507772021"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tpr = tp / (fn + tp)\n",
"tpr"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "5ce0bbcc-ef02-45e8-b465-9f5e105ab797",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.09872922776148582"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fpr = fp / (tn + fp)\n",
"fpr"
]
},
{
"cell_type": "markdown",
"id": "d31259ea-b0c6-4fbf-b22f-410a1a6c81f4",
"metadata": {},
"source": [
"* We now calculated these numbers for the threshold 0.5\n",
"* ROC curves compares results different thresholds"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "a1a5309c-de64-4139-98be-83bea83cdd39",
"metadata": {},
"outputs": [],
"source": [
"scores = []\n",
"thresholds = np.linspace(0,1,101)\n",
"\n",
"for t in thresholds:\n",
" actual_positive = (y_val == 1)\n",
" actual_negative = (y_val == 0)\n",
"\n",
" predict_positive = (y_pred >= t)\n",
" predict_negative = (y_pred < t)\n",
" \n",
" tp = (predict_positive & actual_positive).sum()\n",
" tn = (predict_negative & actual_negative).sum()\n",
" \n",
" fp = (predict_positive & actual_negative).sum()\n",
" fn = (predict_negative & actual_positive).sum()\n",
" \n",
" scores.append((t, tp, fp, fn, tn))"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "daa7a3c0-0f72-4a8e-8230-e081211f168a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(0.0, 386, 1023, 0, 0),\n",
" (0.01, 385, 914, 1, 109),\n",
" (0.02, 384, 830, 2, 193),\n",
" (0.03, 383, 766, 3, 257),\n",
" (0.04, 381, 715, 5, 308),\n",
" (0.05, 379, 683, 7, 340),\n",
" (0.06, 377, 661, 9, 362),\n",
" (0.07, 372, 640, 14, 383),\n",
" (0.08, 371, 613, 15, 410),\n",
" (0.09, 369, 580, 17, 443),\n",
" (0.1, 366, 556, 20, 467),\n",
" (0.11, 365, 528, 21, 495),\n",
" (0.12, 365, 509, 21, 514),\n",
" (0.13, 360, 477, 26, 546),\n",
" (0.14, 355, 453, 31, 570),\n",
" (0.15, 351, 435, 35, 588),\n",
" (0.16, 347, 419, 39, 604),\n",
" (0.17, 346, 401, 40, 622),\n",
" (0.18, 344, 384, 42, 639),\n",
" (0.19, 338, 369, 48, 654),\n",
" (0.2, 333, 356, 53, 667),\n",
" (0.21, 329, 341, 57, 682),\n",
" (0.22, 323, 322, 63, 701),\n",
" (0.23, 320, 313, 66, 710),\n",
" (0.24, 316, 304, 70, 719),\n",
" (0.25, 309, 291, 77, 732),\n",
" (0.26, 304, 281, 82, 742),\n",
" (0.27, 303, 270, 83, 753),\n",
" (0.28, 296, 256, 90, 767),\n",
" (0.29, 291, 245, 95, 778),\n",
" (0.3, 284, 236, 102, 787),\n",
" (0.31, 280, 230, 106, 793),\n",
" (0.32, 278, 226, 108, 797),\n",
" (0.33, 276, 221, 110, 802),\n",
" (0.34, 274, 213, 112, 810),\n",
" (0.35000000000000003, 272, 207, 114, 816),\n",
" (0.36, 267, 201, 119, 822),\n",
" (0.37, 265, 197, 121, 826),\n",
" (0.38, 260, 185, 126, 838),\n",
" (0.39, 252, 179, 134, 844),\n",
" (0.4, 249, 166, 137, 857),\n",
" (0.41000000000000003, 246, 159, 140, 864),\n",
" (0.42, 243, 157, 143, 866),\n",
" (0.43, 241, 150, 145, 873),\n",
" (0.44, 234, 147, 152, 876),\n",
" (0.45, 230, 134, 156, 889),\n",
" (0.46, 224, 125, 162, 898),\n",
" (0.47000000000000003, 218, 120, 168, 903),\n",
" (0.48, 217, 115, 169, 908),\n",
" (0.49, 213, 110, 173, 913),\n",
" (0.5, 210, 101, 176, 922),\n",
" (0.51, 207, 99, 179, 924),\n",
" (0.52, 204, 93, 182, 930),\n",
" (0.53, 196, 91, 190, 932),\n",
" (0.54, 194, 86, 192, 937),\n",
" (0.55, 185, 79, 201, 944),\n",
" (0.56, 182, 76, 204, 947),\n",
" (0.5700000000000001, 176, 68, 210, 955),\n",
" (0.58, 171, 61, 215, 962),\n",
" (0.59, 163, 59, 223, 964),\n",
" (0.6, 151, 53, 235, 970),\n",
" (0.61, 145, 49, 241, 974),\n",
" (0.62, 141, 46, 245, 977),\n",
" (0.63, 133, 40, 253, 983),\n",
" (0.64, 125, 37, 261, 986),\n",
" (0.65, 119, 34, 267, 989),\n",
" (0.66, 114, 31, 272, 992),\n",
" (0.67, 105, 29, 281, 994),\n",
" (0.68, 94, 26, 292, 997),\n",
" (0.6900000000000001, 88, 25, 298, 998),\n",
" (0.7000000000000001, 76, 20, 310, 1003),\n",
" (0.71, 63, 14, 323, 1009),\n",
" (0.72, 57, 11, 329, 1012),\n",
" (0.73, 47, 10, 339, 1013),\n",
" (0.74, 41, 8, 345, 1015),\n",
" (0.75, 33, 7, 353, 1016),\n",
" (0.76, 30, 6, 356, 1017),\n",
" (0.77, 25, 5, 361, 1018),\n",
" (0.78, 19, 3, 367, 1020),\n",
" (0.79, 15, 2, 371, 1021),\n",
" (0.8, 13, 2, 373, 1021),\n",
" (0.81, 6, 0, 380, 1023),\n",
" (0.8200000000000001, 5, 0, 381, 1023),\n",
" (0.8300000000000001, 3, 0, 383, 1023),\n",
" (0.84, 0, 0, 386, 1023),\n",
" (0.85, 0, 0, 386, 1023),\n",
" (0.86, 0, 0, 386, 1023),\n",
" (0.87, 0, 0, 386, 1023),\n",
" (0.88, 0, 0, 386, 1023),\n",
" (0.89, 0, 0, 386, 1023),\n",
" (0.9, 0, 0, 386, 1023),\n",
" (0.91, 0, 0, 386, 1023),\n",
" (0.92, 0, 0, 386, 1023),\n",
" (0.93, 0, 0, 386, 1023),\n",
" (0.9400000000000001, 0, 0, 386, 1023),\n",
" (0.9500000000000001, 0, 0, 386, 1023),\n",
" (0.96, 0, 0, 386, 1023),\n",
" (0.97, 0, 0, 386, 1023),\n",
" (0.98, 0, 0, 386, 1023),\n",
" (0.99, 0, 0, 386, 1023),\n",
" (1.0, 0, 0, 386, 1023)]"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scores"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "3ab7ac85-2d57-4446-b73a-ad948ae0b0e5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>threshold</th>\n",
" <th>tp</th>\n",
" <th>fp</th>\n",
" <th>fn</th>\n",
" <th>tn</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.00</td>\n",
" <td>386</td>\n",
" <td>1023</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.01</td>\n",
" <td>385</td>\n",
" <td>914</td>\n",
" <td>1</td>\n",
" <td>109</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.02</td>\n",
" <td>384</td>\n",
" <td>830</td>\n",
" <td>2</td>\n",
" <td>193</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.03</td>\n",
" <td>383</td>\n",
" <td>766</td>\n",
" <td>3</td>\n",
" <td>257</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.04</td>\n",
" <td>381</td>\n",
" <td>715</td>\n",
" <td>5</td>\n",
" <td>308</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" threshold tp fp fn tn\n",
"0 0.00 386 1023 0 0\n",
"1 0.01 385 914 1 109\n",
"2 0.02 384 830 2 193\n",
"3 0.03 383 766 3 257\n",
"4 0.04 381 715 5 308"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# put scores into data frame\n",
"columns =[\"threshold\", \"tp\", \"fp\", \"fn\", \"tn\"]\n",
"\n",
"df_scores = pd.DataFrame(scores, columns=columns)\n",
"df_scores.head()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "e3edb2d0-cda9-4492-b42f-52a7d9a9e8a4",
"metadata": {},
"outputs": [],
"source": [
"df_scores[\"tpr\"] = df_scores.tp/(df_scores.tp + df_scores.fn)\n",
"df_scores[\"fpr\"] = df_scores.fp/(df_scores.fp + df_scores.tn)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "3261064c-3572-4bec-a305-911c5317ac60",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>threshold</th>\n",
" <th>tp</th>\n",
" <th>fp</th>\n",
" <th>fn</th>\n",
" <th>tn</th>\n",
" <th>tpr</th>\n",
" <th>fpr</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.00</td>\n",
" <td>386</td>\n",
" <td>1023</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.01</td>\n",
" <td>385</td>\n",
" <td>914</td>\n",
" <td>1</td>\n",
" <td>109</td>\n",
" <td>0.997409</td>\n",
" <td>0.893451</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.02</td>\n",
" <td>384</td>\n",
" <td>830</td>\n",
" <td>2</td>\n",
" <td>193</td>\n",
" <td>0.994819</td>\n",
" <td>0.811339</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.03</td>\n",
" <td>383</td>\n",
" <td>766</td>\n",
" <td>3</td>\n",
" <td>257</td>\n",
" <td>0.992228</td>\n",
" <td>0.748778</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.04</td>\n",
" <td>381</td>\n",
" <td>715</td>\n",
" <td>5</td>\n",
" <td>308</td>\n",
" <td>0.987047</td>\n",
" <td>0.698925</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" threshold tp fp fn tn tpr fpr\n",
"0 0.00 386 1023 0 0 1.000000 1.000000\n",
"1 0.01 385 914 1 109 0.997409 0.893451\n",
"2 0.02 384 830 2 193 0.994819 0.811339\n",
"3 0.03 383 766 3 257 0.992228 0.748778\n",
"4 0.04 381 715 5 308 0.987047 0.698925"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_scores.head()"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "d7fdbccb-a4ce-4919-b2db-78c8df848f6b",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxbUlEQVR4nO3deVzVVf7H8de5l1VQkFUBEcR9BcV9SVPLrdQ2NVMry3HMmtbJqWmqqX4zzTQ1ObnkUlaWlmWLZVpp7prihruiooKIiBsgyHZ+f3yxyEivcrlf7r2f5+PBQ7n3y73v79S8+3Lu+Z6jtNYIIYRwfhazAwghhLAPKXQhhHARUuhCCOEipNCFEMJFSKELIYSL8DDrjUNCQnRMTIxZby+EEE5p8+bNp7TWoRU9Z1qhx8TEkJSUZNbbCyGEU1JKHfm952TIRQghXIQUuhBCuAgpdCGEcBGmjaELIYQ9FBUVkZaWRkFBgdlR7MrHx4eoqCg8PT1t/hkpdCGEU0tLS6NmzZrExMSglDI7jl1orcnOziYtLY3Y2Fibf+6qQy5KqXeUUieVUjt/53mllJqslEpRSiUrpdpeQ24hhKiUgoICgoODXabMAZRSBAcHX/NvHbaMoc8B+l3h+f5Ao7KvccC0a0oghBCV5Eplfsn1nNNVC11rvQo4fYVDBgPva8MGIFApVfeak9jo2N4kNs58hG837WXfiRwKi0ur6q2EEMKp2GMMPRI4Vu77tLLHMi4/UCk1DuMqnujo6Ot6s4zUvXRIf48hh+LYphviYVHEhvjROLwmjcL9aRxek8bh/tQP9sPTKpN4hBBVKzs7m969ewNw4sQJrFYroaHGjZzbt2+nTZs2FBcX06xZM9577z1q1KiB1WqlVatWFBcXExsbywcffEBgYGCls9ij0Cv6vaDCXTO01jOAGQCJiYnXtbNGh3btYQO81a8WSbXi2Z+Zw/7MXHYeP8finRlc2q/Dy8NCj0ahDGpdl97NwqjpY/snxUIIYavg4GC2bdsGwAsvvIC/vz9PPvkkAP7+/j8/N3LkSKZPn87jjz+Or6/vz4+PGTOGKVOm8Oyzz1Y6iz0KPQ2oV+77KOC4HV63YrVjQFmIKs0gKiHyV0/lF5ZwMCuX/Zk5JKedY+muE/ywJxMvDws9G4cysHVdejcLx99bJvcIIRyre/fuJCcn/+bxzp07V/j49bBHs30FTFRKzQc6Aue01r8ZbrEbD28IiILsg795ytfLSsvIAFpGBnBb2yj+Nqg5W4+dYdH2DL7dmcF3uzPx9rDQq0lYWbmHUcNLyl0IV/Hiol3sPn7erq/ZPKIWz9/SolKvUVxczLfffku/fr+eX1JSUsKyZcsYO3ZspV7/kqu2mVJqHtATCFFKpQHPA54AWuvpwGJgAJACXADus0uyKwmKg9O/LfTLWSyKdvWDaFc/iL8Nas7mo2f4evtxFu88wZJdJ/DxtPDUzU25v6vrzF8VQlQf+fn5xMfHA8YV+qXivvR4amoq7dq1o2/fvnZ5v6sWutZ6xFWe18BDdkljq+A4SF4AWoONRWyxKNrHBNE+Joi/3dKCTamnmbHqEC99vZuUk7n8fXAL+RBVCCdX2Stpeys/Vl7R4+fOnWPQoEFMmTKFRx55pNLv55wNFhQHF8/Bhezr+nGrRdGpQTCzRicyoWcc8zYeZcw7G9ly9Aw5BUV2DiuEEBULCAhg8uTJvPbaaxQVVb57nHMAOTjO+DP7IPiFXPfLWCyKP/drSoNQf55ZuIPbpq4DICLAhy4NQxjYui5d40Lw8nDO/+4JIaq/hIQE2rRpw/z58xk1alSlXss5Cz2orNBPH4TojpV+uTvaRdG9UQjJaefYn5nDnozzLN11gk83pxHg68mw9vUY16MBIf7elX4vIYTreuGFF371fW5uboXHXf74okWL7PL+zlnoteuDslY40+V6hdfyoW9zH/o2DwfgYnEJq/ef4vNt6cxafYgP1h9hdJf63N42iphgP7lqF0JUO85Z6FZPCIy2aabL9fL2sNKneTh9modzMCuXycsOMGPVId5eeQgPiyImxI8gP6+fj7+hcSh/vCEOi0VmywghzOGchQ7GOLodr9CvJC7UnzeHJ/BE3yZsOXrm57tTcy8aH2JcKCzh30v3sev4Of5zZzy+XlaH5BJCiPKct9CD4uDohmuaulhZ0cE1iA6u8ZvHtdbMXnOYVxbv4djp9cwak0h4LR+HZBJCiEucdyA4OA4KcyH3pNlJUErxQPcGzByVyMGsXAa/tZad6efMjiWEcDPOW+jlZ7pUE32ah/Pp+C5YFNw5fT1Ldp4wO5IQwo04b6EHNzD+dNA4uq2aR9Tii4ldaVKnJuPnbmbysgMUl8ia7UK4MqvVSnx8/M9fqamprFixgoCAABISEmjWrBkvvvgiwK8eb9q06c8rM9qD846hB0SDxaNaXaFfElbTh/njOjHps2Re/34/y/ee5LU729AwzN/saEKIKlDRLf6pqal0796dr7/+mry8POLj4xk0aBDAz4/n5+eTkJDA0KFD6dq1a6VzOO8VutXDWEq3ml2hX+LjaeWNYfH8b0QCqdl5DJi8mpmrDlFSel3LwAshnJifnx/t2rXj4MFf95Wvry/x8fGkp6fb5X2c9wodylZdPGR2it+llOKWNhF0bBDEs5/v5JXFe1iy6wSv3dmG2BA/s+MJ4Xq+nQQndtj3Neu0gv7/vOIh5VdVjI2N5fPPP//V89nZ2WzYsIHnnnuOrKysnx8/c+YMBw4coEePHnaJ6tyFHhwHqasdOnXxeoTV9GHGqHZ8sS2d57/cRf83V3F/11haRwX+vF2eVW5IEsJp/d6qiqtXryYhIQGLxcKkSZNo0aIFK1asYPXq1bRu3Zp9+/YxadIk6tSpY5cczl3oQQ2g6ALkZECtCLPTXJFSiqEJUXSJC+HZz3cwdcUvv3oF+Hpyc4twBraOoEtcsCzjK8T1usqVtKNdGiv/vcf3799Pt27dGDp06M9X+JXh3IUe0sj4M2tftS/0S8Jr+TBrTHtyLxaTctLYLm/DwWwW7zjBJ0lp1Avy5cOxnSq8gUkI4VoaN27MX/7yF1599VXmzZtX6ddz7kvB8JbGn5k7zc1xHfy9PYivF8hdifV4fVg8SX/tw/R72pJTUMyImRs4dvqC2RGFEA4wfvx4Vq1axeHDhyv9Ws5d6H4hUDPC/h+CmMDH00q/lnWZO7YjuReLGT5jA2lnpNSFcAYVLZPbs2fPCodbLn/c19eX9PR0YmNjK53DuQsdjE+gXaDQL2kZGcDcsR3JKShi2NsbOJCZY3YkIYSTcI1Cz9oHRQVmJ7GbVlEBfPhAJy4Wl3Lb1HWs3J919R8SQrg91yh0XQJZe8xOYletogL4cmJXImv7ct+7G5mz9jDGftxCiMu54v83ruecXKPQwaWGXS6JDPTl0z924camYbywaDf3z9lE5nnX+U1ECHvw8fEhOzvbpUpda012djY+Pte2DLdzT1sEqB0LXv4uWehgzIaZMSqR99an8uqSvfR9fSV/H9ySIQmRZkcTolqIiooiLS3tV3dgugIfHx+ioqKu6Wecv9AtFmP6oosWOoDForivayw9m4Tx5ILtPPrxNjLPF/CHG+LMjiaE6Tw9Pe0yQ8QVOP+QC5TNdNkJpa69TG1siB8fj+vELW0i+Me3e5m5qvquYyOEcDzXKfTCHDh7xOwkVc7DauGNu9owsFVdXlm8h1mrD7nU2KEQ4vq5TqGDSw+7lOdhtfDf4fH0b1mHl7/Zwx3T17PmwCkpdiHcnGsUelgzUFa3KXQAT6uF/41I4KUhLTl+Np97Zv/EsBkb2C83Ignhtlyj0D19IaSxWxU6GFfqozrVZ8VTPfn74BaknMxl0OQ1TFtxUDbSEMINuUahg8stAXAtvD2sjO4cw3eP9eDGpmG8umQvd0xfR3buRbOjCSEcyLUK/XwaXDhtdhLThPh7M+2etrw5PJ7dx88z7oPNFBSVmB1LCOEgNhW6UqqfUmqfUipFKTWpgucDlFKLlFLblVK7lFL32T/qVdRtbfx5fKvD37o6UUoxOD6S1++KZ/ORMzz9WbJ8WCqEm7hqoSulrMAUoD/QHBihlGp+2WEPAbu11m2AnsB/lFJeds56ZZHtjA9Gj6xz6NtWVwNb1+Wpm5vw5bbjvLnsgNlxhBAOYMudoh2AFK31IQCl1HxgMLC73DEaqKmUUoA/cBootnPWK/OuCRHxcGStQ9+2OpvQM45DWXn894cDeFotTOgZh6rGe68KISrHliGXSOBYue/Tyh4r7y2gGXAc2AH8SWv9m9s2lVLjlFJJSqmkKll3oX5XSN8MRfn2f20npJTiH7e14tY2Efx76T6eWLCdi8Uypi6Eq7Kl0Cu6pLt8UPZmYBsQAcQDbymlav3mh7SeobVO1FonhoaGXmNUG9TvCiWFkLbJ/q/tpLw8LLw5PJ7H+zZm4ZZ07p75E59tTmNH2jnyC6XchXAltgy5pAH1yn0fhXElXt59wD+18elbilLqMNAU2GiXlLaK7gQoSF0LsT0c+tbVmVKKR3o3okGoH5M+28ETC7YD4GlVvHp7a25re20rugkhqidbCn0T0EgpFQukA8OBuy875ijQG1itlAoHmgCOXznKN9CYvijj6BUa1DqCfi3qkJp9gQOZOby7LpWnP0smItCXTg2CzY4nhKikqw65aK2LgYnAUmAP8InWepdSarxSanzZYS8BXZRSO4BlwNNa61NVFfqKYroZQy7FclNNRTysFhqG+dO/VV1mjk4kOqgG4+du5vCpPLOjCSEqyaZ56FrrxVrrxlrrOK31K2WPTddaTy/7+3Gt9U1a61Za65Za67lVGfqK6neF4gLjw1FxRQG+nrxzb3sUMHbOJk7nFZodSQhRCa5zp+gl9bsYf6bKsIst6gf7MWN0Imln87lt6loOZuWaHUkIcZ1cr9BrBEFYCziyxuwkTqN9TBAfPdCRnIJihk5Zy9oUc0bLhBCV43qFDhDTFY5thJIis5M4jcSYIL54qCt1AnwY/c5GFu/IMDuSEOIauWah1+8KRRcgfYvZSZxKvaAafPbHLiTUC+Sxj7ex9egZsyMJIa6BaxZ6bA9jXZcDS81O4nRq+ngyY3Qi4bV8ePD9JI6dvmB2JCGEjVyz0GsEQXRn2LvY7CROKcjPi3fubU9hcSlj39vE+QIZuhLCGbhmoQM0HQBZe+C04+9vcgUNw/yZdk87DmXlMWjyGn46lG12JCHEVbhuoTcZYPwpV+nXrWvDED56sBMAw2du4MVFu8iRq3Uhqi3XLfSgWGP64j4p9MroEBvEkke7M7pTfd5dm0q7l39g3PtJfLktndyLjl0hWQhxZbas5eK8mg6A1f+BvGzwk7VKrlcNLw9eHNyS29tF8fnWdBbvyOC73Zl4e1jo1SSMga3r0rtZGDW8XPtfJyGqO2XW9mSJiYk6KSmpat8kfQvM7AVDpkH85euJietVWqpJOnKGxTsy+GZHBlk5F/HxtHBj0zAGtorgxqZh+HpZzY4phEtSSm3WWidW+JxLF7rW8HpziGwLwz+s2vdyUyWlmk2pp/kmOYNvd2ZwKrcQX08rvZuFMah1XXo2CcPHU8pdCHu5UqG79u/ISkGT/rB9nrGLkaev2YlcjtWi6NQgmE4Ngnnh1hb8dDibb5IzWLLzBF8nZ9AozJ/ZY9oTHVzD7KhCuDzX/VD0kqYDjLtGD3xndhKXZ7UousSF8MrQVvz0TG/eHtWOkzkXGTJ1LZtST5sdTwiX5/qFHnsDBNaHNW8YQzDCITysFm5uUYcvHupKoK8nI2f+xNQVKWSck/1ehagqrl/oVk/o/gQc3wopP5idxu3Ehvjx+YSudGkYzL+W7KPzP5Zzx7R1fJ18+S6GQojKcv1CB2gzAgLqwYp/ylW6CQJqeDLnvg4sf+IGnujbmLP5RUz8aCv/WLyHklL55yGEvbhHoXt4QffHIT0JDv1odhq31SDUn4d7N2LJn7ozqlN93l51iPFzN5MnNygJYRfuUegA8SOhViSseFWu0k3mYbXw0pCWvHhrC5btyeSe2T9RWFxqdiwhnJ77FLqHN3R7DI5tgIPLzE4jgDFdYpg8IoGtR8/y6pK9ZscRwum5T6EDJIyC4Ibw5cNwQabRVQeDWkcwpnN9Zq85zPe7M82OI4RTc69C9/SB22dBXhYsekSGXqqJZwY2o2VkLZ5csJ30szKtUYjr5V6FDhCRAL2fgz2LYMv7ZqcRgLeHlbdGtKWkVHPvOxtZse8kZi1JIYQzc79CB+j8sHHD0ZJJcOqA2WkEEBPix5SRbblQWMK9727i9mnrWH0gS4pdiGvgnoVuscDQ6eDhAwvuNdZ5Eaa7oXEoPz7Zk1eGtuTEuQJGzd7IXW+vZ23KKSl2IWzgnoUOUCsCbpsBmTuNK3VRLXh5WBjZsT4/PtWTl4a05NjpfEbO+olhb29g/UHZBk+IK3HfQgdo1NeYyrh5DiQvMDuNKMfbw8qoTvVZ8VRPXry1BUdO5zFi5gaGz1jPupRTlModpkL8hmuvh26LkmKYM9C4Uh+3EkIamp1IVKCgqIR5G48ydcVBsnIuUqeWD/1b1eH2tlG0jAwwO54QDuO+G1zY6lw6TOsCdVvD6K+MddRFtVRQVMLSXcZa6yv3ZVFcWsqsMYnc2DTc7GhCOMSVCt29h1wuCYiEG/8Kh1cZ0xlFteXjaWVwfCQzRyey6dk+NI+oxcMfbWVPxnmzowlhOpsKXSnVTym1TymVopSq8BNEpVRPpdQ2pdQupdRK+8Z0gHb3QVgL+O5ZmfXiJAJqeDJrdHv8fTwYO2cTJ88XmB1JCFNdtdCVUlZgCtAfaA6MUEo1v+yYQGAqcKvWugVwp/2jVjGrB/T/J5w9Cuv+Z3YaYaM6AT7MHtOeMxeKGPteEifOSakL92XLFXoHIEVrfUhrXQjMBwZfdszdwEKt9VEArfVJ+8Z0kNge0HwwrH4dzqWZnUbYqGVkAG/dncCBkzn0fWMln21Ok3nrwi3ZUuiRwLFy36eVPVZeY6C2UmqFUmqzUmp0RS+klBqnlEpSSiVlZWVdX+KqdtPLgIYV/zA7ibgGvZuFs+RPPWgSXpMnFmxn3AebyZV11oWbsaXQK5rycfnljwfQDhgI3Aw8p5Rq/Jsf0nqG1jpRa50YGhp6zWEdIjAa2o6G7R/LVbqTiQnx4+M/dOavA5uxfO9J7n93k2yeIdyKLYWeBtQr930UcPmGkGnAEq11ntb6FLAKaGOfiCbo8gigZSzdCVktige6N+C/w+JJOnKa++ds4kKhlLpwD7YU+iagkVIqVinlBQwHvrrsmC+B7kopD6VUDaAjsMe+UR0osB60Hg6b34Pcajo0JK7oljYRvDEsnk2pRqnnF5aYHUmIKnfVQtdaFwMTgaUYJf2J1nqXUmq8Ump82TF7gCVAMrARmKW13ll1sR2g26NQXAA/TTM7ibhOg+MjeWNYPBsPn2bse1LqwvXJnaJX8skYOLgcHtsJPnJ7ubP6fGsaj3+yna5xIcwak4iPp9XsSEJcN7lT9Hp1fxwunof1U81OIiphaEIUr93RhrUHT3H/nE0s3XWCw6fyKJEFvoSL8TA7QLVWtw20vB3WvA4thkBYM7MTiet0e7soSrXmmc93sK5sGV4/LyuP9mnM/d1isVpk/R7h/GTI5WryTsGUDlA7BsZ+Dxb5dd2Z5V0s5sDJXPZn5rBk5wmW7z1Ju/q1ee3ONsSG+JkdT4irkiGXyvALgf7/gvTNsEGGXpydn7cH8fUCuSuxHrPHJPLGsDYcyMyh/5ureHXJXs7kFZodUYjrJoVui5a3Q5OBsPxlyD5odhphJ0ophiZE8f3jN3BT8zpMX3mQbq8u599L93L2ghS7cD4y5GKrnBPG0EtEWxj1uayZ7oL2Z+bw5rIDLN6RgZ+XB/d3jWFstwYE1PA0O5oQP5MhF3uoWQd6/gUO/QgHvjc7jagCjcNrMuXutiz5Uw96NA5h8vIUuv1rOW98v59z+UVmxxPiquQK/VoUF8LUTsYHo39cB1a5cnNlezLO898f9rN0Vya1fDx4oHsDxvVoIPPYhankCt1ePLzgppfg1H5jY2nh0prVrcXboxL55pFudGwQzOvf7+eJBdtlaV5RbUmhX6smAyCmu7G8bv5Zs9MIB2gREcDM0Yn8uV8TvknO4KONR82OJESFpNCvlVLGmukXThuzXoTbGN8jjh6NQ3lx0W7Zw1RUS1Lo1yMiHjqOh00zYetcs9MIB7FYFK/f1YZAX08e+mgL5wvkg1JRvUihX6+bXoYGPWHRo3B0g9lphIOE+Hvz5vAEUk/l0fGVZUz8aAtLdmZQXFJqdjQhpNCvm9UD7pxj7HA0f6SxubRwC53jgvn0j10Y2jaSdQezGT93C5MW7jA7lhBS6JXiWxvu/hhKi+CLCWanEQ7UNro2/ze0FRuf6c24Hg34dHMay/dmmh1LuDkp9MoKaWTccJS6Go6sMzuNcDAPq4UnbmpM43B/nlm4U8bVhamk0O2h7RjwC4OV/zI7iTCBt4eVf9/RhpM5BbzytfPuvCicnxS6PXjVgK6PGMsCHNtodhphgjb1AhnXI46Pk47xSdIxSmXzDGECKXR7SbwfagTLVbobe7RPI1pHBfDnT5MZ+L81LN11Qu4qFQ4lhW4vXn7Q5WFI+d5YO124HR9PKwv/2IXX72pDfmExf/hgM+PnbqawWKY0CseQQren9g8YM1+W/R3kyswteVgt3NY2ih8ev4FJ/ZuydFcmD8/bQpHMUxcOIIVuT941odezcGgFbPvI7DTCRB5WC+NviOP5W5qzdFcmj8zbKqUuqpwUur0ljoXoLrD0L3A+w+w0wmT3dY3luUHN+XbnCf44dwt5F4vNjiRcmBS6vVksMPgtKL4I3zwuQy+Csd1i+fvgFizfm8kd09dz/Gy+2ZGEi5JCrwrBcXDjX2HfYtj5mdlpRDUwunMMs+9tz7HTFxg8ZS3JaWfNjiRckBR6Vek0ASITjcW7Tsg6HwJ6NQlj4YQueFkt3D8niZM5BWZHEi5GCr2qWKww7APwqQUf3gXn0s1OJKqBxuE1efe+9uReLOKxj7dRIjcgCTuSQq9KtSJg5AIozIUP74SCc2YnEtVA4/CavHhrC9amZDNtRYrZcYQLkUKvauEt4K734dQ+Y5ndIvlATMBdifUYHB/B69/v56dD2WbHES5CCt0R4nrBkGmQugbm3w1FMnbq7pRSvDK0FdFBNfjLwh0y9CLsQgrdUVrfBYOnwMEf4eORxrRG4db8vT14ul9TDp3K4+vk42bHES7ApkJXSvVTSu1TSqUopSZd4bj2SqkSpdQd9ovoQhJGwi1vQsoP8MkYKJGbTNzdzS3q0CS8JpOXHZCrdFFpVy10pZQVmAL0B5oDI5RSzX/nuFeBpfYO6VLajYEBr8H+b2Hxk3LjkZuzWBSP9G7Ewaw8vtkhdxaLyrHlCr0DkKK1PqS1LgTmA4MrOO5h4DPgpB3zuaYOD0LXR2Hzu7D2v2anESbr37IOjcL8+d+yA7KOuqgUWwo9EjhW7vu0ssd+ppSKBIYC06/0QkqpcUqpJKVUUlZW1rVmdS29n4eWt8MPL8COT81OI0xksSge7t2IAydzWbxTrtLF9bOl0FUFj11+GfFf4GmtdcmVXkhrPUNrnai1TgwNDbUxoouyWIyZL/W7wufjIWWZ2YmEiQa2qktcqB+vfLOHY6cvmB1HOClbCj0NqFfu+yjg8o/kE4H5SqlU4A5gqlJqiD0CujQPbxj+EYQ2hY/vgWObzE4kTGK1KCaPSOBCYQnDZ2wg7YyUurh2thT6JqCRUipWKeUFDAe+Kn+A1jpWax2jtY4BPgUmaK2/sHdYl+QbCPd8Bv7h8OEdkLnb7ETCJC0iApg7tiM5BUWMmLmBdFmVUVyjqxa61roYmIgxe2UP8InWepdSarxSanxVB3QLNcNh9Bfg6QsfDIGTsnO8u2oVFcDcBzpy9kIRQ6es5ce9MsdA2E6ZtYltYmKiTkpKMuW9q62Te+H9wVBSCKM+h4h4sxMJk+w+fp5HP97K/sxc7mwXxXO3NKeWj6fZsUQ1oJTarLVOrOg5uVO0OglrCvctNjacfu8WOPqT2YmESZpH1GLRw92Y0DOOz7akMWTKWnJltyNxFVLo1U1wHNy/BPxCjav1XZ+bnUiYxNvDyp/7NeW9+zuQeiqPZxbuwKzfqIVzkEKvjgKi4P6lULc1LLgXfvwHlMoGw+6qe6NQHuvTmK+2H+eTpGNX/wHhtqTQqyv/UBizCOJHwsp/wqf3yiqNbmxCr4Z0bRjM81/tYt+JHLPjiGpKCr068/A2Vmjs+xLs/hLmDYPCPLNTCRNYLYo3hsXj7+3JQx9t4UKhjKeL35JCr+6Ugq6PGHeVHl4FH9wmOx+5qbCaPrw5PJ6DWbk8/+Uus+OIakgK3VnE3w13vAPpSfDerXBRfu12R10bhjCxV0MWbE5j4ZY0s+OIakYK3Zm0GArDPoQTO+CzB6H0ikvnCBf1p96N6BATxF+/2MnBrFyz44hqRArd2TTpB/3+aaynvvwls9MIE3hYLUwekYC3h4UJc7dw9kKh2ZFENSGF7ow6PAiJ98OaN2D7x2anESaoE+DD/0a05fCpPEbN3si5/CKzI4lqQArdGSkF/f8FMd3hy4eMNdVlTN3tdGsUwvRRbdl74jyjZ/8kpS6k0J2W1ROGfQCt7jCu1Ce3hS0fyLi6m7mxaTjTRrZjd4ZR6qdyZfNxdyaF7sx8a8PQ6fDAcqgdA19NhBk9IXWt2cmEA/VpbpT6vswcBr+1Vm48cmNS6K4gqh2M/Q5unw0XTsOcAfDJaMjLNjuZcJA+zcP55A+dKSop5fZp62TZXTclhe4qlDKGXyZugl5/hX1LYGYv2TDDjbSOCuTLiV2pH1yDB99PYme63IDmbqTQXY1XDbjhKbjvWyi+CLP7wt7FZqcSDlI3wJePHuhEbT8vnlywncJiWdTNnUihu6qodjDuRwhpBPPvhsVPyUwYNxFQw5NXhrRk74kcpq5IMTuOcCApdFdWK8K4Uu8wDjbOhCkd5WrdTdzUog63tongreUp7Mk4b3Yc4SBS6K7O0xcG/Ase+AF8AmH+CFg4Thb4cgMv3NqCwBqePPXpdopLZOjFHUihu4uoRPjDSuj5F9jxKUzrCqlrzE4lqlCQnxcv3NqCnennWbgl3ew4wgGk0N2J1RN6TjKmOFo9Yc4gWPsmyLZmLmtgq7okRAfy+vf7KSiSm85cnRS6O4pKhPFroMUQ+P5v8O3Tcoepi1JKMalfU06cL+DdtalmxxFVTArdXXn5we3vQKeHYOPbxt6lRflmpxJVoGODYHo3DWPqihRZmdHFSaG7M4sF+v0f3PQK7PnKWDYgY7vZqUQV+HO/puReLGbqioNmRxFVSApdQJeJcM9CyD8LM3vD6v/IEIyLaVKnJre3jWLOulS2HD1jdhxRRaTQhaFhb5iwHpoOgGV/h7d7GHuYCpfxdL+mRAT4MGb2RrZKqbskKXTxixpBcOd7cOccKDgP790C80fC8W0yE8YFhNb0Zt64TgT5ezF69ka2HztrdiRhZ1Lo4teUMvYunbgRbnwODv4IM26A6d1gwzQ4e8zshKIS6gb4Mu9BY62Xe2b/xPGz8kG4K1HapCuvxMREnZSUZMp7i2uQfwZ2fgZb58LxrcZjtWOM3ZIa9jG+vP1NjSiu3ZHsPPq8vpJh7evx8pBWZscR10AptVlrnVjRcx6ODiOcjG9taP+A8ZW1z7hiP7zKmBWz9QPw8IG4GyFhFDTpb1zhi2qvfrAfdybW45NNaTzUqyF1A3zNjiTswKYhF6VUP6XUPqVUilJqUgXPj1RKJZd9rVNKtbF/VGG60CbQaTyM+AieOgRjvoa2Y4wx9vkjjGmP+7+T8XYnMaFnHBrNdJnK6DKuWuhKKSswBegPNAdGKKWaX3bYYeAGrXVr4CVghr2DimrG6gGx3Y2Fvx7dAYOnQP5p+OhOY52Y9VMh75TZKcUVRNWuwR3topi36RiZ5wvMjiPswJYr9A5Aitb6kNa6EJgPDC5/gNZ6ndb60jyoDUCUfWOKas3qAQn3wMNb4Nb/gYc3LP0L/KcpfHwP7F8KJcVmpxQVmNCzIaWlmmlyle4SbBlDjwTKT21IAzpe4fixwLcVPaGUGgeMA4iOjrYxonAaVk9oO9r4ytwN2z6E7fNhzyLwrwPNb4XYHlC/qzFFUpiuXlANbmsbybyNR5nQM46wWj5mRxKVYMsVekWfclU4SKqU6oVR6E9X9LzWeobWOlFrnRgaGmp7SuF8wpvDza/AE3th+EcQ2Ra2fGBcsf+rAXw0DHKzzE4pgId6NaSopJR316WaHUVUki2FngbUK/d9FHD88oOUUq2BWcBgrbVsNy8MVk9oOhBGzINJR+G+JdDjSWO2zNvd4ch6sxO6vfrBfvRrWYe5G46Qe1GGxpyZLYW+CWiklIpVSnkBw4Gvyh+glIoGFgKjtNb77R9TuAQPL6jfGW78q7GDkocPzBkIy1+G87+5RhAONK5HHDkFxczfeNTsKKISrlroWutiYCKwFNgDfKK13qWUGq+UGl922N+AYGCqUmqbUkruGBJXVre1sYNS88Gw6t/wRguYewek/GB2MrcUXy+QDrFBvLPmMEWyXZ3TkjtFhfmyD8K2j2D7PDifDj3+bGyVZ5GVKRxp2Z5Mxr6XxH+HxTMkIdLsOOJ3XOlOUfl/jDBfcBz0fg4e2WpMf1z1L1gwGi7mmp3MrfRqEkajMH+mrzyIWRd6onKk0EX14eENt74FN/8D9n5jLOG7+T0okpteHMFiUTzYowF7T+TwjmxX55RkLRdRvSgFnSdAWFP47m+w6BFjffbWwyCkEQRGQ1gzqBVhdlKXdFtCJD/szuSlr3fjaVWM7hxjdiRxDWQMXVRfWkPqalg/BVKWQWlR2RMKmgyAzg9B/S6yIJidFRaXMuHDLfywJ5OXh7Tknk71zY4kyrnSGLoUunAOpSWQkwFnj8LB5bBptrF2TGgzqNce6rSG6E5QR5aCtYfC4lL+OHczy/ae5N1729OraZjZkUQZKXThegovQPJ82PUFnEg21m0HuOFpuGGSzJCxg4vFJQycvIbC4lK+e6wHPp5WsyMJZJaLcEVeNSDxfhjzFfz5MDy2C+JHwspXYcEYKMwzO6HT8/aw8sItLTh6+gIzVx0yO46wgRS6cH5KQUCUsYTvzf8He7+G2TdD+mazkzm9bo1CGNCqDlNWpJB25oLZccRVSKEL16GU8UHp3QsgNxNm3gifj5dlBSrp2YHG9gcvf73H5CTiaqTQhetp1Ace3gzdHjP2Q52cAPPuhm3z4MJps9M5nchAXyb2asiSXSf4clu62XHEFcg8dOGafGpBnxeg3b3G7kl7FsG+b0BZIaYbNLsFmg6CWnXNTuoUHuzRgJX7s3j8k+14WS30byX/u1VHMstFuIfSUji+FfYugj1fQ/YB4/HIRGg2CJrdaixBIH5X7sVixryzke3HzvLW3W3p17KO2ZHckkxbFOJyWftgz1dGuWdsMx5rPgT6vgi1Y0wMVr3lFBQx5p2NJKed49372tO9kWxU42hS6EJcydljsHUurJts3MDUeQI07GssM1ArAiwy/7q88wVF3DltPadyL7L4T90Jl23rHEoKXQhbnEuHZS9C8se/PGb1goZ9jFUgG91k7MAkSDmZwy3/W0ubegF8+EAnrBZZfsFRpNCFuBZnjxlj7GeOGEMzuxYa0yBrhBh7owZGQ2B9Yx2ZiLZue1fqp5vTeHLBdv7UuxGP9W1sdhy3caVCl1kuQlwusJ7xdclNLxs7Ke1YAKf2w7GNUHDWeK5mhLFnasJIiEgwJa5Z7mgXxbqDp5i8/AABvp6M7BSNt4cMT5lJrtCFuB552UbJ7/nKWAmyOB+iuxg3NjW+2W2GZvIuFvPAe0msP5RN3QAfJvRqyLDEenh5uOdvLY4gQy5CVKWC87D1A9gwHc4dBau3sWZ73dbGdMiGfVx6iV+tNWtSTvHG9/vZcvQsHWKCmD6qHUF+XmZHc0lS6EI4QkkxHFgKR9dDRjJkbDeGZiITodczEHejyxf7l9uO8/RnyYTV8uadMe1pFF7T7FguRwpdCDMUF8K2D2HVa3A+DSLbQeeJxlW71XU/vtp69AwPvr+Zi0UlvD2qHV0ahpgdyaVIoQthpuKLxjz39W/B6UMQUM9YdqB2fWO2TESCyy1BkH42n/vf3cSR03m8M6a9lLodSaELUR2UlsD+pbBhqrG0b1HZcrTKYgzHxI80ttbzdI0bdbJzL3L3zJ84cjqPd+/tQOe4YLMjuQQpdCGqG63hQjacSYX9S4yVIM+ngYcvRHeEmO7GPPfwlsZCY07qVO5FRszYQNqZfKbd05aeTWQru8qSQheiuistgUMr4MB3cHg1nNz1y3NBDSC6M7QZDvW7Od2NTFk5F7ln1k/sy8xhZMdonhnQDD9v1/0MoapJoQvhbPJOGcMyGclwYjscWgkXzxtj7i2GGuPudVtD7VinmDlTUFTCf77bx6w1h4kM9OWpm5vQp1m4FPt1kEIXwtkVXjC21ts6F1LXgC4xHveuBXVaQZ3Wxp91W0No02p7Y1NS6mn+/Gkyh07l4e1h4camYYy/IY429QLNjuY0pNCFcCVF+XByD5xILruC3wGZO3/5kNXqBSFNfplFExRrjMXXaQne5s8LLy3VJB05wzfJx/k6OYPCklI+n9CFhmHmZ3MGUuhCuLrSEsg+WFby2yFrr7G42NmjxrIEAChjYbEaweATAH6h0LC3sVSBb21TYqefzWfwW2vx9bLwxYSuBPt7m5LDmUihC+GutDY2yT6xwyj7U/sh/wwUnDPKPjcTLB7GrJo2I4yt+bxqODTitmNnGfb2elpGBvDhAx3x8ZQFvq5ECl0I8VuXtuXb8xXs/sKYQuldC1oMMRYaq9saQho7ZDx+8Y4MJny4hfrBNWgZGUDjsJq0q1+bTg2C8LA616yeqlbpQldK9QPeBKzALK31Py97XpU9PwC4ANyrtd5ypdeUQheiGikthaPrYOuHsPtLKMozHrd4Qo0gY4jGJxACIo1x+cB6xvrwPgHgG/jL8961rntZgy+3pbNo+3H2Z+Zy7MwFtIYgPy/6tazD7W2jaFffnGGh6qZSha6UsgL7gb5AGrAJGKG13l3umAHAwxiF3hF4U2vd8UqvK4UuRDVVUgzZKcYwzcndxg1QBecg/7Sxq9O5Y1BS+Ps/7+X/S8HXrFP24Ww0+IX9Uv7WcmPlPgHG8+XukM27WMzqA6f4ZkcGy/ZkcqGwhO6NQnisb2PaRrt3sVd2g4sOQIrW+lDZi80HBgO7yx0zGHhfG/912KCUClRK1dVaZ1QyuxDC0aweENbU+KpIaakx9p5/uqzozxpz5C/9veBc2d/PwPl0Y1gn//TV39e/jlH4gB/Qr+yrNExzNr+IM8cKKZmlSXWBtdZPxN1Jp5HP2/11bSn0SOBYue/TMK7Cr3ZMJPCrQldKjQPGAURHR19rViFEdWCxGIuJXcuCYhdzjCv9/LPGksIlxWVPaLhwGs4eMWblFOb89u2AIKBWqeZo9gVO51/htwMn4VEzvGpe14ZjKroN7fJxGluOQWs9A5gBxpCLDe8thHAF3jWNr0qMlngADewWyDXZ8rtLGlBug0WigOPXcYwQQogqZEuhbwIaKaVilVJewHDgq8uO+QoYrQydgHMyfi6EEI511SEXrXWxUmoisBRj2uI7WutdSqnxZc9PBxZjzHBJwZi2eF/VRRZCCFERmyaMaq0XY5R2+ceml/u7Bh6ybzQhhBDXwvnn/wghhACk0IUQwmVIoQshhIuQQhdCCBdh2mqLSqks4Mh1/ngIcMqOcZyBnLN7kHN2D5U55/pa69CKnjCt0CtDKZX0e4vTuCo5Z/cg5+wequqcZchFCCFchBS6EEK4CGct9BlmBzCBnLN7kHN2D1Vyzk45hi6EEOK3nPUKXQghxGWk0IUQwkVU60JXSvVTSu1TSqUopSZV8LxSSk0uez5ZKdXWjJz2ZMM5jyw712Sl1DqlVBszctrT1c653HHtlVIlSqk7HJmvKthyzkqpnkqpbUqpXUqplY7OaG82/LsdoJRapJTaXnbOTr1qq1LqHaXUSaXUzt953v79pbWull8YS/UexNikxAvYDjS/7JgBwLcYOyZ1An4yO7cDzrkLULvs7/3d4ZzLHbccY9XPO8zO7YB/zoEY+/ZGl30fZnZuB5zzM8CrZX8PBU4DXmZnr8Q59wDaAjt/53m791d1vkL/eXNqrXUhcGlz6vJ+3pxaa70BCFRKXcNGh9XOVc9Za71Oa32m7NsNGLtDOTNb/jkDPAx8Bpx0ZLgqYss53w0s1FofBdBaO/t523LOGqiplFKAP0ahF+OktNarMM7h99i9v6pzof/extPXeowzudbzGYvxX3hndtVzVkpFAkOB6bgGW/45NwZqK6VWKKU2K6VGOyxd1bDlnN8CmmFsX7kD+JPWutQx8Uxh9/6yaYMLk9htc2onYvP5KKV6YRR6typNVPVsOef/Ak9rrUuMizenZ8s5ewDtgN6AL7BeKbVBa72/qsNVEVvO+WZgG3AjEAd8r5RarbU+X8XZzGL3/qrOhe6Om1PbdD5KqdbALKC/1jrbQdmqii3nnAjMLyvzEGCAUqpYa/2FQxLan63/bp/SWucBeUqpVUAbwFkL3ZZzvg/4pzYGmFOUUoeBpsBGx0R0OLv3V3UecnHHzamves5KqWhgITDKia/WyrvqOWutY7XWMVrrGOBTYIITlznY9u/2l0B3pZSHUqoG0BHY4+Cc9mTLOR/F+I0EpVQ40AQ45NCUjmX3/qq2V+jaDTentvGc/wYEA1PLrliLtROvVGfjObsUW85Za71HKbUESAZKgVla6wqnvzkDG/85vwTMUUrtwBiOeFpr7bTL6iql5gE9gRClVBrwPOAJVddfcuu/EEK4iOo85CKEEOIaSKELIYSLkEIXQggXIYUuhBAuQgpdCCFchBS6EEK4CCl0IYRwEf8PuWY4UW8vZFMAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(df_scores[\"threshold\"], df_scores[\"tpr\"], label=\"TPR\")\n",
"plt.plot(df_scores[\"threshold\"], df_scores[\"fpr\"], label=\"FPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "a33eeb0b-feaf-484b-ac7f-61c10c35e663",
"metadata": {},
"source": [
"* threshold=0 is the dumy model, that predicts everyone is churning. In this case bose TPR and FPR are 1.\n",
"* We want to minimize FPR and maximiz TPR"
]
},
{
"cell_type": "markdown",
"id": "c7b2fe23-289d-416f-b088-ab60cf51b4c5",
"metadata": {
"tags": []
},
"source": [
"### Random Model"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "cba5f791-3afc-4bc1-aa73-2d9666fefa72",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([4.17022005e-01, 7.20324493e-01, 1.14374817e-04, ...,\n",
" 7.73916250e-01, 3.34276405e-01, 8.89982208e-02])"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.random.seed(1)\n",
"y_rand = np.random.uniform(0, 1, size=len(y_val))\n",
"y_rand"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "0f1c8e4c-1382-4793-8b01-2c5e5e71b3c2",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5017743080198722"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Accuracy of random model\n",
"((y_rand >= 0.5) == y_val).mean()"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "55ab9059-13b3-4ba3-81cd-0f4f08d8a503",
"metadata": {},
"outputs": [],
"source": [
"# write tpr, fpr into dtaframe\n",
"# as previous code, but in a function\n",
"def tpr_fpr_dataframe(y_val, y_pred):\n",
" scores = []\n",
" thresholds = np.linspace(0,1,101)\n",
"\n",
" for t in thresholds:\n",
" actual_positive = (y_val == 1)\n",
" actual_negative = (y_val == 0)\n",
"\n",
" predict_positive = (y_pred >= t)\n",
" predict_negative = (y_pred < t)\n",
" \n",
" tp = (predict_positive & actual_positive).sum()\n",
" tn = (predict_negative & actual_negative).sum()\n",
" \n",
" fp = (predict_positive & actual_negative).sum()\n",
" fn = (predict_negative & actual_positive).sum()\n",
" \n",
" scores.append((t, tp, fp, fn, tn))\n",
" \n",
" columns =[\"threshold\", \"tp\", \"fp\", \"fn\", \"tn\"]\n",
" df_scores = pd.DataFrame(scores, columns=columns)\n",
" \n",
" df_scores[\"tpr\"] = df_scores.tp/(df_scores.tp + df_scores.fn)\n",
" df_scores[\"fpr\"] = df_scores.fp/(df_scores.fp + df_scores.tn)\n",
" \n",
" return df_scores"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "9c0b8e0a-f96b-4af2-8912-8ff759666655",
"metadata": {},
"outputs": [],
"source": [
"df_rand = tpr_fpr_dataframe(y_val, y_rand)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "3eba1737-0699-4790-91a4-931c9cf43d48",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAr8UlEQVR4nO3dd3QU5dvG8e+dRkIICYRQAyRBeoeI9CLSQUCUKghIUxBERbD3guJLRwRElCqiIB3pvQWB0KS3AFIChhJKyvP+segPMcACm0x29/6ck3PcncnuNSd4ZfLszPOIMQallFLOz8PqAEoppRxDC10ppVyEFrpSSrkILXSllHIRWuhKKeUivKx642zZspmwsDCr3l4ppZzSli1bzhljQlLaZlmhh4WFERUVZdXbK6WUUxKRo3fapkMuSinlIrTQlVLKRWihK6WUi7BsDF0ppRwhISGBmJgYrl27ZnUUh/L19SU0NBRvb2+7v0cLXSnl1GJiYggICCAsLAwRsTqOQxhjiI2NJSYmhvDwcLu/755DLiIyXkTOiMjOO2wXERkmIgdEJFpEyt1HbqWUeijXrl0jODjYZcocQEQIDg6+77867BlDnwDUv8v2BkDBm1/dgK/vK4FSSj0kVyrzvz3IMd2z0I0xq4Dzd9mlKfCDsdkABIlIrvtOYqcLZ0+yYVQ3Lv11LrXeQimlnJIjrnLJAxy/5XHMzef+Q0S6iUiUiESdPXv2gd7s4KZ5PHp6OteHRLJn6STQ+dyVUhaKjY2lTJkylClThpw5c5InT55/HosIZcqUoUSJEjzzzDPEx8cD4Onp+c/zTZo04a+//nJIFkcUekp/F6TYssaYMcaYSGNMZEhIineu3lNko67sazqbCx5ZKLq6J7sHN+H8hbv9AaGUUqknODiYbdu2sW3bNnr06EHfvn3/eezv78+2bdvYuXMnPj4+jB49GgA/P79/ns+aNSsjR450SBZHFHoMkPeWx6HASQe87h0VLVedvP03sDS0J4Xj1hA1pDVfLtzDX/E3UvNtlVLqgVWrVo0DBw785/lKlSpx4sQJh7yHIy5bnA30EpFpwGNAnDHmlANe9658M2SgdpdPObsoC3XXf8zva4ZSdX1z+jcowrOP5XPJD0mUUnf3wZxd7D550aGvWSx3Zt5rUvyhXiMxMZEFCxZQv/6/ry9JSkpi6dKlPP/88w/1+n+7Z6GLyFSgJpBNRGKA9wBvAGPMaGA+0BA4AMQDnRySzE4hdV+DuJ303/MjJrgM78xKZNHOP/m8RUlCs2RMyyhKKfUvV69epUyZMoDtDP3v4v77+SNHjlC+fHnq1KnjkPe7Z6EbY9rcY7sBejokzYMQgaYjkbN/MODyQErU+Y7+Ky9Qf8hqulWPoFOVMAJ87b/TSinlvB72TNrR/h4rv9PzcXFxNG7cmJEjR9K7d++Hfj/XmMslQyZoNRnx8KRJVEeWt/SlYkQw/7d4H9W+WM7I5Qe4eiPJ6pRKKfUvgYGBDBs2jEGDBpGQkPDQr+cahQ6Q7RF4fjH4ZSXHrFaMq3CK2b2qUC5fFr5ctJdGw1az5egFq1MqpdS/lC1bltKlSzNt2rSHfi0xFl3HHRkZaVJlgYsrsTClJZzYApV7Qc03WXM0nv4/R3Mq7ipdq0fwSp1CZPDydPx7K6XS3J49eyhatKjVMVJFSscmIluMMZEp7e86Z+h/8w+G5+ZAuQ6wbjiMqkhV2cbCPlVp9Whevll5iH4/RWPVLzKllEotrlfoAD4Z4clh0HEeeHrDpBYEDC/KZ5ff5ZeCi9gfvZ4hS/ZbnVIppRzKtafPDasKPdbC9qlwIgpORVP2zBoWZEhgzapJrE/sRaV6rcHDNX+vKaXci2sXOoC3L0R2sn0BcvUCiZsnUHTFCII3vsDVPV/iV/0lKNXadmavlFJOyv1OTf2y4FW9L559o/kkQ18OXwTm9oXBxWHPHKvTKaXUA3O/Qr8pKMCftl370VY+52W/T0kMyg8/dYKDy6yOppRSD8RtCx0gPJs/o9tHMu9iOC/KOyRnKwTTnoXjm62OppRyIn9Ph/v315EjR1ixYgWBgYGULVuWokWL8sEHHwD86/kiRYrw2muvOSyHWxc6QMWIYD5tXpLfDl2jckwvTiQGcOW75qxfPofkZL20USl1b3/fyv/3V1hYGGCbv2Xr1q1ERUUxadIktmzZ8q/nt27dyty5c1m7dq1Dcrh9oQM8E5mX7ztX4Knq5RieZxCXkjNQaeWzbPn0cdavWqTFrpR6KP7+/pQvX56DBw/+63k/Pz/KlCmTrqbPdQk1CoVQo1AIUISka7XYNef/KLRrDIHLWnJwZQSeEdXJX74eElFTr4ZRKr1aMAD+3OHY18xZEhp8ftddbp1VMTw8nJkzZ/5re2xsLBs2bOCdd97h1tXaLly4wP79+6levbpDomqhp8DTNxPFn3mXxEZ9iJ47nMQ/FlJ832Rk/wQSfQLxqvA8VOgGmVNt6VSllBO506yKq1evpmzZsnh4eDBgwACKFy/OihUrWL16NaVKlWLv3r0MGDCAnDlzOiSHFvpdeGUMpFTLt0lMepPZWw6zasls6l+bT901Q/BYNxyKNoGyz0JETfDQuWGUstw9zqTTWrVq1Zg7d+4dn9+3bx9Vq1alefPm/5zhPwwdQ7eDl6cHT1UowAd9ezE5/8fUuP4Vm0JaYA4th0lPwZCStnljkhKtjqqUciKFChXijTfeYODAgQ55PS30+xDo5834jo9S87EKtDz6JK0DJrC32nBM8CPw29swtiac3Gp1TKWUE+nRowerVq3i8OHDD/1arjd9bhqZHnWcr37by+mL16kQloUvih8lbOP7cOUMVO4Nj79tmxhMKZWqdPrc/9Ez9AfUMjIvK/vV4v0mxTgSG0/dhUHMrT7TNqa+dgh83wQunrQ6plLKjWihPwRfb086Vgnnt77VKZ8/C71+OcygDL1Ibj4WTkXD6GpwaIXVMZVSbkIL3QGCMvrwfecKtH40LyOWH+C1Pwphui4D/2ww8SnYMBp0QQ2lUo0rLljzIMekhe4gPl4efPZUSV5+oiC/bD3B4O0e0GUJFKoPC/vDnD6QeMPqmEq5HF9fX2JjY12q1I0xxMbG4uvre1/fp9ehO5CI0Kd2QU5cuMqwpfuJyOZPs1aTYPknsHqQ7Q62hl9CaIqfZyilHkBoaCgxMTH/ugPTFfj6+hIaGnpf36OF7mAiwifNS3LsfDyvz4gmNIsfkbXfsd0+vOB1GFcbSrWC2u9C4P39sJRS/+Xt7U14eLjVMdIFHXJJBT5eHnzTvjx5svjRbeIWjsXGQ/Fm8NIWqPYq7JoFQ8vAzB62D0+VUsoBtNBTSVBGH8Z3fJSkZEOnCZuIu5oAGQJsZ+YvRUFkZ9g9G76pBpOfgctnrI6slHJyWuipKDybP9+0L8+x8/H0nPw7CUnJtg1B+aDhF/DKLqj9HhxebbvE8Yhj5kRWSrknvVM0DfwUdZx+M6IpFRpIsL8PANkDfKlfMidVCmTDJ3YPTO8A5w9D7Xegch/w0N+1Sqn/utudovqhaBp4JjIvcVcTmL39JLFXbJcuRh29wI9Rxwn086Ze8Rw8+fhPVN71IR5L3odjG6DZ15Axq7XBlVJORc/QLXI9MYk1+88xL/oUv+0+zeXriWTx8+Lj0I00PDEMCcgFLSdAnvJWR1VKpSN3O0PXQk8HriUksWrfWeZEn2Je9ElqZTrOSO+h+F4/B02GQpm2VkdUSqUTDz05l4jUF5G9InJARAaksD1QROaIyHYR2SUinR42tDvx9fakbvGcDG9Tll9erMIR3yJUPP8eB/1KwqwXYNFbkJxkdUylVDp3z0IXEU9gJNAAKAa0EZFit+3WE9htjCkN1AS+EhEfB2d1C2XyBjGvdzVaVi9NvXN9WJixCawfAZNawInfrY6nlErH7PlQtAJwwBhzCEBEpgFNgd237GOAABERIBNwHtDlex6Qr7cnbzYsSqnQQPpM96GLXx5eOT4Jz7G1IF9lqNQTCjfUK2GUUv9iTyPkAY7f8jjm5nO3GgEUBU4CO4A+xpjk219IRLqJSJSIRLnavAupoXGp3PzYvRI/mjpUuj6c34v2w8Qdhx/bwTfV4Y95OoujUuof9hS6pPDc7S1SD9gG5AbKACNEJPN/vsmYMcaYSGNMZEhIyH1GdU9l8gYxu1cVCoTm5qmtZemceSx/1RsOCVdgWlsY+zjEHrQ6plIqHbCn0GOAvLc8DsV2Jn6rTsAvxuYAcBgo4piIKneQH5O7PMaHTYuz4UgcVRfmYHDhScQ3GAYXDttK/dBKq2MqpSxmT6FvBgqKSPjNDzpbA7Nv2+cYUBtARHIAhYFDjgzq7jw8hA6Vwlj4cjWqFczG0OVHeGxBTr4r/h3JATlhYnPYNNbqmEopC92z0I0xiUAvYBGwB5hujNklIj1EpMfN3T4CKovIDmAp0N8Ycy61Qruz/MH+fP1seeb3rkblAsF8sCaeZtfeJy5vLZj/Giz9SMfVlXJTemORk1u17yz9f47m3KWrTM8znbJnf+VquW74NfkCJKWPP5RSzuyhbyxS6Vf1QiEsfLk6zcrmpfnxloxLbIDf72OY/VELZm0+6FLLciml7k7P0F3I6YvX+OPURQLWf0G5I2P502RhabZnqfNsP7JnCbQ6nlLKAfQM3U3kyOxLjcLZKddxEEkd5mCyhNEudjhJQ8uyad53mOT/3BqglHIhWuguyjOiOrn6LOdEk6lc9QqkwuaX2f5VY86fOmx1NKVUKtFCd2Ui5CnfkHz9N7Iu4mUKX96MzzeVmDH5G3aeiNPxdaVcjI6hu5FD+3dhfupM2PW9vJvYiTVBT9KoZC4alcpFsVyZEb0qRql0T+dDV/9z4wo3pj2Hz6HF/Jq5Da+ea0xishCRzZ+OVcJo9WheMnh5Wp1SKXUH+qGo+h8ff3zaTYNyHWh6cSp7c77H9Mg95MiYzLu/7qLmlyuYuOHo/xa0Vko5DS10d+TpBU2GQYtv8czgT4WdHzHl0vOsilxLsYCrvDNrJ01HrGXPqYtWJ1VK3QcdcnF3xsDRdbB+JOydj/Hw4mTexvSKeZyd17LRp3ZBetQogJen/u5XKj2425CLPQtcKFcmAmFVbF+xB5GNo8mzdRK/mLksy96cl3+ry7bjcYxqVw4fLy11pdIz/T9U/U9wAWj4JfTehpRuTe0LP7EpoB/F9n3Nez8s0HF1pdI5LXT1XwE5oOkI6LYCv3zleMV7Bp8cbceBQU+QeHiN1emUUnegha7uLHcZaP8L9IlmW4HuBMYfxuP7xhya+hrJCdetTqeUuo0Wurq3LPkp12EgO5stZr53HSL2juXw5xVZvWG93m2qVDqiha7sVrfsIzR4YzrrHxtBcNI5Ci5oQ+chP7N492ktdqXSAb1sUT2QxFM7SR7fgDOJ/jS7+i6Zgm1TCDQqmZuiuQJ0GgGlUoneKaoczitXCXw6/EwerziWZB9C4aBkRq88RMNhq3lz5g6r4ynllrTQ1YPLWwFpNYmgy4f45mJPousfoWOFXEzddJwlu09bnU4pt6OFrh7OI7Wh4zzIEo7/sjd473A7umfdypszdxAXn2B1OqXciha6enj5HoNO86H9LCQgB2/Ef8nL10bx2ZytVidTyq1ooSvHEIECteD5JVClD209l9JuV1dWbthkdTKl3IYWunIsTy+o8yEJLScT5nmWCgsa8ePQfuw4Fmt1MqVcnl62qFLNlbNHOTOtF+Gxq9iZHMZovy7s8CwOIpTLl4WPm5XAP4POD6fU/dAVi5R1jOHq9p8x814nY0Isp33ysi6gPp+dLEdwzrx8+1wkuYP8rE6plNPQ69CVdUTwK/M0GV+LhidHkCNnKM1jx7I2U39KnF9M05Fr2XrsgtUplXIJWugqbWTIBOXaQ+eF8OJGvHMU4UsZyifJQ+k8eglDluzT6XmVekha6CrtZS8CnRZArbepY9axyu81Li8fQqsRS3XZO6Uego6hK2ud3AZL3odDyzlHEMOTniKkZg961Cyoy94plQIdQ1fpV+4y0GEWdFpIYN7ifOA5ntIrOtNlxGz2nb5kdTqlnIpdhS4i9UVkr4gcEJEBd9inpohsE5FdIrLSsTGVy8tfCe/O86DJUCr5HGTohZ6MHD6Q16Zv41hsvNXplHIK9xxyERFPYB9QB4gBNgNtjDG7b9knCFgH1DfGHBOR7MaYM3d7XR1yUXcUe5CEGV3xPrWFZcnleDexM9UiyzCgQREC/bytTqeUpR52yKUCcMAYc8gYcwOYBjS9bZ+2wC/GmGMA9ypzpe4quADeXX6Dup9Qy2cPS31fJ+PvY2j4f8tYsVf/aSl1J/YUeh7g+C2PY24+d6tCQBYRWSEiW0SkQ0ovJCLdRCRKRKLOnj37YImVe/D0gsq9kJ4byRBRmXe8fmBK0quM/34cb83coZc4KpUCewo9paVnbh+n8QLKA42AesA7IlLoP99kzBhjTKQxJjIkJOS+wyo3lCU/tJsBraeSN9CbH3wGUvP3PvSfuo5ELXWl/sWeQo8B8t7yOBQ4mcI+C40xV4wx54BVQGnHRFRuTwSKNMSj50ao8yG1vbbRYV9v3pq6WktdqVvYU+ibgYIiEi4iPkBrYPZt+/wKVBMRLxHJCDwG7HFsVOX2vHygSh88Wk2kpOcxOu7rxXtTlnMjUUtdKbCj0I0xiUAvYBG2kp5ujNklIj1EpMfNffYAC4FoYBMwzhizM/ViK7dWpBGez07nEa8zdN/fnfEjPuKvi3rNulJ6p6hyXsc2EvdTTwIv7ee8BEGF7mSt85rtTF4pF6V3iirXlO8xAl/ZzL66P7DHhJF140AOf1WT1VHbuJ6YZHU6pdKcnqErl3D8fDwrZ35D8+Ofc9140T/5Jc7kqEqhHAEUz52Z5uVC9aYk5RJ0gQvlNm6c3seNKc/iH7ePr7O8xvhLFTl3+ToBvl50qRpBp6phZPbVYlfOS4dclNvwyVGITD1XIOHVefGvr4h68gLzelelUkQwg5fso+rnyxi2dD+XriVYHVUph9MzdOWabsTDlJZwdC20GAclWrDzRBxDluxjyZ4zBGX0pkvVcFpG5iV7Zl+r0yplNx1yUe7p+mWY/Awc3wCVX4IaA8AnI9ExfzFkyX6W/XEGEagQlpXGpXPTKjIvPl76R6tK37TQlfu6fhkW9oetkyBLGDQeDAUeB+DAmUvMjT7F3OhTHDhzmSeK5mBUu3Ja6ipd00JX6vBqmNMHzh+ER+pArTcgT/l/Nk9cf4R3ft1FnWI5GNlWS12lX/qhqFLh1eCFdfDE+3AiCsY+DlNaQ+xBANpXCuODJ4uzePdpXpr6u87mqJySFrpyH96+ULUv9ImGx9+2fWA6qhKsGgSJN3iuchjvNi7Gol2n6fjdJuLi9UoY5Vy00JX78c0M1ftBr81QuD4s+wjG1IAja+hcNZxBz5Rm0+HzNB+1lsPnrlidVim7aaEr9xWQE1r+AK2nwrWLMKER/NiepyMSmdylIhfib9Bs5Fq2H//L6qRK2UULXakiDW1n67XeggNLYMSjVNj5AfPaZCeznxfPfx9FzAVdqFqlf1roSgH4ZIQar8NLW6BMW9g+jdyTa7Aw2zDCEg/y/IQovbtUpXta6ErdKnNuaDIU+u6CWm/hfy6an+QNnjo/hlcnr9cVklS6poWuVEr8s9nO2HtFIWXb0d1zDm8f7cyECWNI1lJX6ZQWulJ3kzErPDkcnptLJv+MdDnen/2D65J8WldYVOmPFrpS9givRtZXo1ge3pecl3Zhvq6CWfimbWoBpdIJLXSl7OXlQ80O7zG5wix+TKyBbBiJGVUR9i+2OplSgBa6UvdFRHihYQViqn7G09ff5ew1T5j8NMzsAdfirI6n3JwWulL3SUToV68wkdUbUTXuA5bn6IiJ/hG+rgpH1lodT7kxLXSlHoCI0L9+YTpWL0yno3XpF/AlcTcMZkIjWP0VWDSLqXJvWuhKPSAR4Y0GRfioWQn2eheh0oUPmJ1UCZZ+yDfvPUfht+fTfWIUZy5dszqqchM6H7pSDnI09gqLdp4kctcnlDszkw3ZWvDcny3w8/Hmw6YlaFIqFyJidUzl5HSBC6XSkjHw29uwfgRXc1di0OX6jD9TgPolcvNRsxJky5TB6oTKiekCF0qlJRGo+zE0HITfpaO8c/E9fs/6Np5/zKHe4FUs2HHK6oTKRWmhK5UaRKBCV3g5Gp4aR5aATIzwGsz73hPoM3kjL03dyoUrN6xOqVyMFrpSqcnTG0o9A91WQMWeNLk2l9XZBhK9M5o6g1exePdpqxMqF6KFrlRa8PSG+p9Cy4nkSIhhacD71PLdT9cfonhx8hYOnLlkdULlArTQlUpLxZ6Ersvx8g/mi/i3GVdiFyv2nqXO4FX0nrqVqCPndd519cD0KhelrHD1L5jRGQ4u5Vr5HgzxfI7v1x/jakISALkDfaleKIQBDYoQlNHH2qwqXXnoq1xEpL6I7BWRAyIy4C77PSoiSSLy9IOGVcot+AVB2+lQoTu+W0YzIGEk616vwdgOkfSrV5jyYVmZsSWGOoNXsXSPjrMr+3jdawcR8QRGAnWAGGCziMw2xuxOYb+BwKLUCKqUy/H0ggYDwTcQVn1BloSr1Gn+DXWK5QBgV40IXp2+nee/j6JVZF4+bFacDF6eFodW6Zk9Z+gVgAPGmEPGmBvANKBpCvu9BPwMnHFgPqVcmwg8/hY88QHs/Bl+fBZu2BakLp47kNm9qvJizQL8GHWctmM3cu7ydYsDq/TMnkLPAxy/5XHMzef+ISJ5gObA6Lu9kIh0E5EoEYk6e/bs/WZVynVVfRkafQX7FsH3TeBKLAA+Xh68Xr8II9qWZeeJOJqNXMveP/WKGJUyewo9pcknbv8kdQjQ3xiTdLcXMsaMMcZEGmMiQ0JC7IyolJt4tAu0mgind8K3deD84X82NS6Vmx+7V+J6YjLNRq7l+3VHSE7WGR3Vv9lT6DFA3lsehwInb9snEpgmIkeAp4FRItLMEQGVcitFm0CHXyE+FsbXg9P/+6iqTN4g5vSqSoXwrLw3exftxm3k+Pl4C8Oq9MaeQt8MFBSRcBHxAVoDs2/dwRgTbowJM8aEATOAF40xsxwdVim3kK8idF4ECExoCCe3/rMpZ6AvEzo9ysAWJdlxIo4n/m8lH83dzdlLOrau7Ch0Y0wi0Avb1St7gOnGmF0i0kNEeqR2QKXcUvYi0HkB+ATA90/C0fX/bBIRWj2aj0V9q9OkdG6+W3uYal8s462ZO5i++Thbj13gyvVEC8Mrq+iNRUqlZ3Ex8ENTuHAUar0JVfqAx78vXTx87grDl+5n3o5TXE9MBiCDlwftK+ane40ChATodL2uROdDV8qZxZ+HuS/D7l8hX2VoPhqy5P/PbknJhmPn49l3+hKLdv3JrK0n8PHyoHOVcF6tWxhPD11cwxVooSvl7IyB7dNgfj/AwOPv2Kbn9bjzjUaHzl5myJL9zN5+klfrFOKl2gXTLq9KNbrAhVLOTgTKtIEX19k+NF3Y33Zp45877vgtESGZGNq6DE+Wzs3gJfvYeCg2DQMrK2ihK+VMgvJBuxnQ4lvbuProqjC5JRxaaTuLv42I8OlTJcmXNSO9p20lVu80dWla6Eo5GxEo+TT02gw134ATW+CHJ+HrKrBhtG3M/RaZMngxom05LlxJ4JXp2/UKGBemY+hKObuEa7BjOmz+Fk5tAw9vKNIIKr4IeSvYfgEAkzce5a2ZO8nq70P36hG0r5SfjD73nJ9PpTP6oahS7uLPnbBtMmybAtf+gjzloXJvKNYURNhy9AJDluxj9f5zhARkYFq3ihQIyWR1anUftNCVcjc3rthKfcPXcP4gFG4ITYZBJtscSlFHztP1hyhyBvox88XK+HrrtLzOQq9yUcrd+PjbLmvsFQX1P4cDS+HrSrB3AQCRYVn5qmVp9py6yMfzdt/jxZSz0EJXypV5eEDFF6DbCsiUE6a2hqUfQXIyjxfJQbfqEUzacIx50aesTqocQAtdKXeQoxh0XQrlOsDqQTC9PVy/TL96hSmTN4gBP0ezZv85q1Oqh6SFrpS78MpgG0evPxD2zodv6+J98Tgj2pYlJHMGnv12I2/N3KGXNToxLXSl3IkIVOxhuznpYgyMrUXoxW3M712NrtXCmbLpGPWHrmL9Qb2r1BlpoSvljh6pDV2WgV8W+P5JfKMn8lbDovzUvRKeIrQZu4H3Z+8i/oaerTsTLXSl3FW2R6DLUgivDnP6wKQWRGY8w/w+1ehYOYwJ647QYOhqFu48pcvdOQktdKXcmV8QtJ0O9T6DmCj4ujIZFw/g/ap+TO1aEU8Rekz6nUbD17Bo159Ydd+Kso/eWKSUsrkSCys+hajxYJIhf1WSSrdjTlJFhqw4ypHYeErkyczLtQtRu2h2RHR+dSvonaJKKfvFnYDtU2HrJLhwGALzklT1VWZRk6HLj3DsfDylQgNp91g+6hXPSVBGH6sTuxUtdKXU/TPGdofpis/gRBQE5SepfGfmUYPBG+M4fO4KXh5ClUey8V6TYkTonDBpQgtdKfXgjIH9i2H1V3B8A4gnpmAd9pfsx88x/kzbdJw8QX782qsK3p76sVxq07lclFIPTgQK1YXnF9nmhqnSGzm+iUJzmvFGgeMMbFGK3acuMnrFQauTuj0tdKWU/bIVhCfehx6rIWs4TGlJ/bgfaVwyJ8OW7Wfvn5esTujWtNCVUvcvMBQ6L7TNs774XQb5fkuWDB70m7GdxKRkq9O5LS10pdSD8fGHZyZA9X747pjMnJCR7I85TfNR6xi3+hAn/7pqdUK3ox+KKqUeXtR4zLxXic1cjJc9+rPmlG3BjHL5gmhcKjcNS+YiZ6CvxSFdg34oqpRKXZGdkVaTyXblIJOu92Vjixv0q1eYawnJfDh3NxU/W8qXi/6wOqXL00JXSjlGkYbQbTlkzkWOeR3peWkY83uUZdmrNXiqbB5GLj/IlI3HrE7p0rTQlVKOk72obcKvKi/D7z/A902I8L/BF0+XombhEN75dacupJGKtNCVUo7llQHqfABtpsLpnTChMV5XzzG8TVkeCcnEC5O3cOCMXt6YGrTQlVKpo3AD20yOFw7Ddw0IuHKMbztGksHLk04TNhN7+brVCV2OXYUuIvVFZK+IHBCRASlsbyci0Te/1olIacdHVUo5nQK1oP1MuHwGRkQS+lt3ptSHMxev0X3iFq4lJFmd0KXcs9BFxBMYCTQAigFtRKTYbbsdBmoYY0oBHwFjHB1UKeWk8lWEnhuhSh84vJJCc1uwIeRzchxfwBszftc51h3InjP0CsABY8whY8wNYBrQ9NYdjDHrjDEXbj7cAIQ6NqZSyqllzm2bMqDvbmg4iCxyiZE+w3j1j9bMnjhYV0RyEHsKPQ9w/JbHMTefu5PngQUpbRCRbiISJSJRZ8+etT+lUso1ZMgEFbpCry2Y1lMw/tlpeugDlg3rytVrN6xO5/TsKfSUliVJ8depiNTCVuj9U9pujBljjIk0xkSGhITYn1Ip5Vo8PJAijQh9dRU7Q1vzxF8/sfur+pz880+rkzk1ewo9Bsh7y+NQ4OTtO4lIKWAc0NQYE+uYeEopVyae3pTo8g27y39IqRvb8P26PDO/eoGfV23hwhU9Y79f9hT6ZqCgiISLiA/QGph96w4ikg/4BWhvjNnn+JhKKVdWrEkfYlvN5XxwOZpemkqTpXWY80UHBs2P1mK/D1732sEYkygivYBFgCcw3hizS0R63Nw+GngXCAZG3Vw4NvFOk8copVRKcharDMXmYGIPcnnxl3T4YyrRG/6g1YZXKFe6DEVyBlAoZwAl8gSS2dfb6rjpks62qJRKn/6YR9IvPbiemMxHyZ358VoFkvHA38eT5yqH0bVaBFn83W+Bal1TVCnlnM4fhp86wqltJAXm48gjHfj6YmV+3vkX/j5edK0WwQs1C+Dj5T43vev0uUop55Q1HLoug5YT8cycmwJbPmbQ2RdY0rUQ1QpmY/CSfTw5Yg27TsZZnTRd0EJXSqVvHp5Q7EnbItXPzYH4WAosaM/XT4UzrkMksVdu0HTEWr76bS+XriVYndZSWuhKKecRXh1aT4HzB2HyMzxRwJ/FfavTuFQuhi87QLUvljNy+QEuX0+0OqkltNCVUs4logY8/R2c/B0mP03Q9ZMMaV2WOb2qUj5fFr5ctJcnvlrJobOXrU6a5rTQlVLOp2hjeGos/LkDRlaEtcMomcufbzs+ys8vVCYxOZk2Yzdw+NwVq5OmKS10pZRzKvm0bRbHArVg8Tswpgb8MZ/y+YKY3KUiCUmGNmM2cMSNSl0LXSnlvAJDbWPqLX+AhHiY1gbG1qLwlc1M6foY1xOTaD1mg9tcBaOFrpRybiJQrCn03AxNR0J8LExsTpFD3zO162OIwDOj17N492mrk6Y6LXSllGvw9IKyz9qKvXhz+O1timz7jF9frETB7JnoNjGKMasOWp0yVd1zLhellHIq3r7QYjwE5IYNI8ked5xpbb/ktYV/8un8P4i/kcTLTxSyOmWq0EJXSrkeDw+o/ykE5oHF7+F3aAXDq75KgGdVhizZj4cIvWsXtDqlw2mhK6VcV6WeUKg+/PYOHss+4LOgfJSOaMPbi5Pw9BB61nrE6oQOpWPoSinXFlwA2kyBDr8ifllpc3IgGwIGsG/xt4xavt/qdA6lha6Ucg8RNaHbCmg9lWxZszDUZxT5lvXk26XbrU7mMFroSin3IQJFGiLdV5NU+30aeG6m1sqWzJif4rr2TkcLXSnlfjw88KzWF9NhNlm9E2iysT17Jr8O1517/hctdKWU2/KKqIZ/73VsyViFovu/4caQcrBtKli08M/D0kJXSrk178CcFH9pBi9l/Jx9VzPBrB4w+Rm45Hx3lmqhK6XcXmBGb157vgMd5FOG+nQj+fBqGFUR9syxOtp90UJXSikgf7A/Y56rwLgbT9Doxiec88oJPz4LMzrD5TNWx7OLFrpSSt0UGZaVxX1rkCOiFJXODmBapvYk7ZpNwtDyxK39Nt2PrWuhK6XULXIG+vJdx0f5pEU5htxoTp1rn/H79dwELn6F44MfJ/ls+r0ZSYxFv3EiIyNNVFSUJe+tlFL2unDlBvtPX+TI4tHUOzmSjB4JUP11vKv2tk0ElsZEZIsxJjKlbXqGrpRSd5HF34cKEdl4pttbzK46i98Sy+G98hMShpSGzeMg8brVEf+hha6UUnYQEdrXeQyfNhN53rzLtkuZYd6rJAwpC8s+hpPbLB9j1yEXpZS6Txeu3GDsqoPsWz+bTvxKJY89eJAMQfmgyTDbOqep5G5DLlroSin1gGIvX2fM6kPMXreDasmbecX/N7Inn8bjuTmQ99FUeU8dQ1dKqVQQnCkDbzQoypz+TQmq0pmnrw7gWEJm4r9rzsl9W9I8jxa6Uko9pGyZMvBmw6L88npz5pQaxaUkLzwnP8UbY2fy85YYLl5LSJMcWuhKKeUg2QN8eanFE3g9N4sAb3j7xAss/nkskR8tYdzqQ6n+/nYVuojUF5G9InJARAaksF1EZNjN7dEiUs7xUZVSyjkER5QhY681ZMxTnNE+QxgZPJ0v5kWneqnfs9BFxBMYCTQAigFtRKTYbbs1AAre/OoGfO3gnEop5VyC8iKdFsBjL1Dn4i9s8n+F8ws/Y9Kyran2lvacoVcADhhjDhljbgDTgKa37dMU+MHYbACCRCSXg7MqpZRz8fKBBp9Dh1/JnL80r3tPp8XKumya8mGqvJ09hZ4HOH7L45ibz93vPohINxGJEpGos2fP3m9WpZRyThE18egwk4Tua9kW9AQZQ/Knytt42bGPpPDc7Rev27MPxpgxwBiwXYdux3srpZTL8M5Vgkp9p6ba69tzhh4D5L3lcShw8gH2UUoplYrsKfTNQEERCRcRH6A1MPu2fWYDHW5e7VIRiDPGnHJwVqWUUndxzyEXY0yiiPQCFgGewHhjzC4R6XFz+2hgPtAQOADEA51SL7JSSqmU2DOGjjFmPrbSvvW50bf8twF6OjaaUkqp+6F3iiqllIvQQldKKRehha6UUi5CC10ppVyEZQtciMhZ4OgDfns24JwD4zgDPWb3oMfsHh7mmPMbY0JS2mBZoT8MEYm604odrkqP2T3oMbuH1DpmHXJRSikXoYWulFIuwlkLfYzVASygx+we9JjdQ6ocs1OOoSullPovZz1DV0opdRstdKWUchHputDdcXFqO4653c1jjRaRdSJS2oqcjnSvY75lv0dFJElEnk7LfKnBnmMWkZoisk1EdonIyrTO6Gh2/NsOFJE5IrL95jE79aytIjJeRM6IyM47bHd8fxlj0uUXtql6DwIRgA+wHSh22z4NgQXYVkyqCGy0OncaHHNlIMvN/27gDsd8y37LsM36+bTVudPg5xwE7Aby3Xyc3ercaXDMbwIDb/53CHAe8LE6+0Mcc3WgHLDzDtsd3l/p+QzdHRenvucxG2PWGWMu3Hy4AdvqUM7Mnp8zwEvAz8CZtAyXSuw55rbAL8aYYwDGGGc/bnuO2QABIiJAJmyFnpi2MR3HGLMK2zHcicP7Kz0XusMWp3Yi93s8z2P7De/M7nnMIpIHaA6MxjXY83MuBGQRkRUiskVEOqRZutRhzzGPAIpiW75yB9DHGJOcNvEs4fD+smuBC4s4bHFqJ2L38YhILWyFXjVVE6U+e455CNDfGJNkO3lzevYcsxdQHqgN+AHrRWSDMWZfaodLJfYccz1gG/A4UABYLCKrjTEXUzmbVRzeX+m50N1xcWq7jkdESgHjgAbGmNg0ypZa7DnmSGDazTLPBjQUkURjzKw0Seh49v7bPmeMuQJcEZFVQGnAWQvdnmPuBHxubAPMB0TkMFAE2JQ2EdOcw/srPQ+5uOPi1Pc8ZhHJB/wCtHfis7Vb3fOYjTHhxpgwY0wYMAN40YnLHOz7t/0rUE1EvEQkI/AYsCeNczqSPcd8DNtfJIhIDqAwcChNU6Yth/dXuj1DN264OLWdx/wuEAyMunnGmmiceKY6O4/ZpdhzzMaYPSKyEIgGkoFxxpgUL39zBnb+nD8CJojIDmzDEf2NMU47ra6ITAVqAtlEJAZ4D/CG1OsvvfVfKaVcRHoeclFKKXUftNCVUspFaKErpZSL0EJXSikXoYWulFIuQgtdKaVchBa6Ukq5iP8HSS0XKI+t3/8AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(df_rand[\"threshold\"], df_rand[\"tpr\"], label=\"TPR\")\n",
"plt.plot(df_rand[\"threshold\"], df_rand[\"fpr\"], label=\"FPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "c42d9e17-ad71-4533-a804-21257274a465",
"metadata": {},
"source": [
"### Ideal Model\n",
"* All predictions are correct"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "bcf089d9-59b2-461b-8a66-b000b10c50d1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1023, 386)"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"num_neg = (y_val == 0).sum()\n",
"num_pos = (y_val == 1).sum()\n",
"num_neg, num_pos"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "ae00074b-f951-49d7-9af4-f76a9bb8d202",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 0, 0, ..., 1, 1, 1])"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# create ideal validation set\n",
"y_ideal = np.repeat([0,1],[num_neg, num_pos])\n",
"y_ideal"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "15496b90-a744-4770-96bf-d5b0e6de3a12",
"metadata": {},
"outputs": [],
"source": [
"# create predictions (numbers between 0 and 1)\n",
"y_ideal_pred = np.linspace(0, 1, len(y_val))"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "55bb59b9-0adf-45f8-bfd8-163140b8544e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# accuracy of ideal model\n",
"# 72.6 of the customers are not churning\n",
"((y_ideal_pred >= 0.726) == y_ideal).mean()"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "a1d8936d-d4e7-41c9-a7a1-4472ab2b53a5",
"metadata": {},
"outputs": [],
"source": [
"df_ideal = tpr_fpr_dataframe(y_ideal, y_ideal_pred)"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "81c84263-c5ad-4212-9fde-c4905145d7dd",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>threshold</th>\n",
" <th>tp</th>\n",
" <th>fp</th>\n",
" <th>fn</th>\n",
" <th>tn</th>\n",
" <th>tpr</th>\n",
" <th>fpr</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.00</td>\n",
" <td>386</td>\n",
" <td>1023</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1.0</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.01</td>\n",
" <td>386</td>\n",
" <td>1008</td>\n",
" <td>0</td>\n",
" <td>15</td>\n",
" <td>1.0</td>\n",
" <td>0.985337</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.02</td>\n",
" <td>386</td>\n",
" <td>994</td>\n",
" <td>0</td>\n",
" <td>29</td>\n",
" <td>1.0</td>\n",
" <td>0.971652</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.03</td>\n",
" <td>386</td>\n",
" <td>980</td>\n",
" <td>0</td>\n",
" <td>43</td>\n",
" <td>1.0</td>\n",
" <td>0.957967</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.04</td>\n",
" <td>386</td>\n",
" <td>966</td>\n",
" <td>0</td>\n",
" <td>57</td>\n",
" <td>1.0</td>\n",
" <td>0.944282</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" threshold tp fp fn tn tpr fpr\n",
"0 0.00 386 1023 0 0 1.0 1.000000\n",
"1 0.01 386 1008 0 15 1.0 0.985337\n",
"2 0.02 386 994 0 29 1.0 0.971652\n",
"3 0.03 386 980 0 43 1.0 0.957967\n",
"4 0.04 386 966 0 57 1.0 0.944282"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_ideal.head()"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "a3ca2a6c-640e-4a8c-bfef-0f41910db7c4",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmCUlEQVR4nO3deXxV9Z3/8dcnCSFhC1vYsl0QEVD2sKVgsW6AWFBRduJWh7HWmTrOVKft1P4682s7P3+/zs9R6zgubCogoiLu2lqtgBAEQcQl7GEn7HsSvvPHTSCEQG6Se++5y/v5eOTxIPee3PM5gm8O53zO52vOOUREJPoleF2AiIgEhwJdRCRGKNBFRGKEAl1EJEYo0EVEYkSSVztu3bq18/l8Xu1eRCQqrVixYq9zLr269zwLdJ/PR0FBgVe7FxGJSma2+ULv6ZKLiEiMUKCLiMQIBbqISIzw7Bq6iEgwlJSUUFRUxIkTJ7wuJahSUlLIzMykQYMGAf+MAl1EolpRURFNmzbF5/NhZl6XExTOOYqLiykqKqJjx44B/1yNl1zM7Dkz221mX17gfTOzx8ys0MxWm1nfWtQtIlIvJ06coFWrVjET5gBmRqtWrWr9r45ArqFPB4Zf5P0RwKXlX/cAf6xVBSIi9RRLYV6hLsdU4yUX59zHZua7yCajgZnOP4d3qZk1N7P2zrkdta4mAIUbN3Hovd/ySc6PKUtoGIpdiEi5hg0SGd8/i1ZN9P9aNAjGNfQMYGul74vKXzsv0M3sHvxn8WRnZ9dpZ4fW/Ym+O+ZwbNsa7in5B46TUqfPEZGaOQdzlm/hufz+XNq2qdflRKTi4mKuvvpqAHbu3EliYiLp6f4HOb/44gt69epFaWkp3bp1Y8aMGTRq1IjExER69OhBaWkpHTt2ZNasWTRv3rzetVggC1yUn6Evcs5dUc17bwK/dc79tfz7D4F/cs6tuNhn5ubmujo/KbrqJXj9XsgcAJPmQUpa3T5HRC5q1dYD3D2jgJMlZTw+qS/f71LtE+eeWrduHd26dfO6DAAeeeQRmjRpwoMPPghAkyZNOHLkCACTJk2iX79+PPDAA+e8np+fT5cuXfj5z39+3udVd2xmtsI5l1vd/oPRh14EZFX6PhPYHoTPvbDeE2Ds87BtBcy4EY7sCenuROJV76zmLLzve2S2bMQdzy9j8fq9XpcUtYYOHUphYeF5rw8ePJht27YFZR/BuOSyELjPzOYAA4GDobp+fo7Lx0CDRjBvCjx7LUx+BVpdEvLdisSbDs1TmT9tMFc9+hFPf7yBvEtae13SBf36jbV8tf1QUD+ze4dm/OrGy+v1GaWlpbz99tsMH35uf0lZWRkffvghd911V70+v0IgbYsvAUuAy8ysyMzuMrNpZjatfJO3gA1AIfDfwL1BqSwQXa6D/DfgxEF49jr/GbuIBF3jhklMGpjDR9/sYePeo16XEzWOHz9O7969yc3NJTs7+0xwV7zeqlUr9u3bx7XXXhuU/QXS5TKhhvcd8OOgVFMXWQPgrvdh9s0wfRTcOsMf9CISVBMGZvH4n79j5pJN9T5jDZVIqys1NZVVq1Zd8PWDBw8yatQonnjiCe6///567y82Zrm07uwP9daXwkvj4fOZXlckEnPaNE1hZI/2zC8o4ujJUq/LiQlpaWk89thjPProo5SUlNT782Ij0AGatoXb34ROw2DhT+Cj3/l7rkQkaKYO9nH4ZCkLVgbnJp5Anz596NWrF3PmzKn3Z8XWLJeGTWHiXFh4P3z0Wzi0DW74AyTG1mGKeKVvdnN6ZKQxc/EmJg/MjsknNOvjkUceOef7itbEqqq+/sYbbwRl/7Fzhl4hsQGMeRKGPui/9DJnIpzSTRyRYDAz8vN8fLf7CEvWF3tdjlQRe4EOYAZX/xJG/QEK3/ffLFWvukhQjOrZnpaNk5m+eJPXpUgVsRnoFXLvhHEvwO51/l714vVeVyQS9VLK57t8sG4XRfuPeV2OVBLbgQ7QdSTkL1SvukgQTR6UA8DspVs8rkQqi/1Ah7O96smN/Zdfvn3P64pEolqH5qlc170dc5Zv4URJmdflSLn4CHRQr7pIkOXn+ThwrISFq0I7ukkCFz+BDupVFwmiQZ1aclnbpkxfvIlAprbGssTERHr37n3ma9OmTXz00UekpaXRp08funXrxq9//WuAc17v2rXrmcmMwRBfgQ5ne9V7TfT3qr9xP5TpqTeR2jIzpubl8NWOQxRs3u91OZ6qeJS/4svn8wH+CYsrV66koKCA2bNns2LFinNeX7lyJYsWLeLTTz8NSh3xF+igXnWRILmpTwZNGybxwtLNXpcS0Ro3bky/fv1Yv/7cTrvU1FR69+4dUeNzo1NFr3paBrz5D/6bpRPnQZPIG+AvEqkaJScxpk8Gcwu28sixUzRvlOxtQW8/BDvXBPcz2/WAEb+76CYV0xMBOnbsyKuvvnrO+8XFxSxdupRf/vKX7Nlz9pmY/fv3891333HllVcGpdT4PEOvTL3qIvUyfkAWp0pPs+Dz+J3vUvmSS+Uw/+STT+jTpw/XXXcdDz30EJdffvmZ13v27Em7du0YNWoU7dq1C0od8XuGXlnXkf656i/e5u9VnzgPMvt5XZVIVLi8Qxq9MtOYs3wLd3zP5+18lxrOpMNt6NChLFq06IKvf/vttwwZMoSbbrrpzBl+fegMvUJW/7O96jNGwbfvel2RSNQYPyCbb3cd4fMt8X1ztLa6dOnCww8/zO9///ugfJ4CvbJzetUnwIoZXlckEhVu7NWBxsmJvLRsq9elRJ1p06bx8ccfs3Hjxnp/lgK9qsq96m/cD3/+rXrVRWrQpGESP+zdgUWrt3PoRP0Xaog21Y3JHTZsWLWXW6q+npqayrZt2+jYsWO961CgV6eiV733JPjL7/wPIalXXeSixvfP5kTJaV7T4heeUaBfSGIDGP0EXPmPsHKWetVFatAzM82/+MWSzXH/5KhXFOgXYwY/+IXmqosEwMyYOjiHwt1HWBzmxS9i8S+QuhyTAj0Q6lUXCciNvTqEffGLlJQUiouLYyrUnXMUFxeTkpJSq59TH3qgqvaqT5oHGepVF6msYvGLp/6ynq37jpHVslHI95mZmUlRUdE5T2DGgpSUFDIzM2v1M+bV32q5ubmuoKDAk33Xy95CmH0zHN0Dt06HLtd7XZFIRNl24DhDf/8nfnRlJx4e0c3rcmKOma1wzuVW954uudRW685w9wfqVRe5gIzyxS/mLt+qxS/CTIFeF03anNurrrnqIufQ4hfeUKDXVdW56ov+Hk7rbEQEtPiFVxTo9VExV33IA7BiOrz6N1AWf0/JiVRlZuTn+bT4RZgp0OvLDK75FVz9L7DmZXj5dig96XVVIp4b06cDzVKSmBHGFsZ4p0APlqH/ACP+Hb5e5G9tPHHI64pEPNUoOYnbcrN458ud7Dp0wuty4oICPZgG/g2M+SNs/ASmj4TDO72uSMRTUwbnUOaclqgLk4AC3cyGm9k3ZlZoZg9V836amb1hZl+Y2VozuyP4pUaJ3hP9C2QUb4BnroU933hdkYhnclo15qrL2vDisi2cLFXTQKjVGOhmlgg8AYwAugMTzKx7lc1+DHzlnOsFDAP+r5l5vLighy69Bu54E0pP+J8q3bLU64pEPJOf52PvkVO8vUb/Yg21QM7QBwCFzrkNzrlTwBxgdJVtHNDU/GtPNQH2AfE9b7ZDH7jrPWjUCmaOhnVveF2RiCeGdm5Np9aNmbFkk9elxLxAAj0DqLwMSVH5a5U9DnQDtgNrgL9zzp2u+kFmdo+ZFZhZQazNXahWy47+FZDaXgFzp8Cy//a6IpGwS0gwpgzOYeWWA6wuOuB1OTEtkECvbsXXqk8KXA+sAjoAvYHHzazZeT/k3NPOuVznXG56enotS41SjVv5h3pdNgLeehA+eERPlUrcGdsvk8bJiWGdwhiPAgn0IiCr0veZ+M/EK7sDWOD8CoGNQNfglBgDkhvBbbOg3x3w1z/Aq9Og9JTXVYmETdOUBtzSL5NFX+yg+Iie0wiVQAJ9OXCpmXUsv9E5HlhYZZstwNUAZtYWuAzYEMxCo15ikn+hjB/8AlbPgRdvVa+6xJWpg3M4VXaaOcu1kHSo1BjozrlS4D7gXWAdMM85t9bMppnZtPLNfgPkmdka4EPgZ865vaEqOmqZ+Ze0G/2ketUl7nRu05QhnVsze+lmSsvOu8UmQaB56F757gOYN9XfBTN5PqRf5nVFIiH3/le7+NHMAv44qS8jerT3upyopHnokehMr/px9apL3PhB1zZktkjVzdEQUaB7qUMff1ujetUlTiQmGFMG5fDZxn18vVP3kIJNge419apLnBnXP4uGSQnMWKz5LsGmQI8EVXvV3/+VetUlZjVvlMyY3hm8tnIbB49p/YBgUqBHisq96p/+h3rVJabl5/k4XlLGyyvUwhhMCvRIol51iRPdOzRjgK8lM5dspuy0/jUaLAr0SFO5V33TX9WrLjFral4OW/Yd46NvdntdSsxQoEeqPpP8i1BrrrrEqOsvb0e7ZilqYQwiBXok66y56hK7GiQmMGlgNp98t5f1e454XU5MUKBHug594O73oXFr9apLzBk/IJvkxARmLVELYzAo0KNBCx/c+R6066FedYkp6U0bckPP9sxfUcSRk/G9Jk4wKNCjReNWMHWh5qpLzMnP83HkZCkLPi/yupSop0CPJufNVf8b9apL1Oud1ZxemWnMWLwJr4YFxgoFerSp6FW/6heweq561SUm5Of5WL/nKH8t1NTt+lCgRyMz+H6luerPj4RDO7yuSqTObujZntZNkpmhFsZ6UaBHsz6TYOI82LcBnlWvukSvhkmJTBiQzYdf72brvmNelxO1FOjR7tIqveqbl3hdkUidTBqYQ4IZs5aqhbGuFOixoPJc9Vlj1KsuUaldWgrDL2/H3OVbOX6qzOtyopICPVZUzFWv6FX/7GmvKxKptfw8HwePl/Daqm1elxKVFOixpHKv+tv/6J+rflqL8Ur06O9rQdd2TdXCWEcK9FhT0auee6d/rvprmqsu0cPMuD3Px9c7D7Ns4z6vy4k6CvRYlJgEN/w/+MEvz/aqnzzsdVUiARndO4O01AbMWLLJ61KijgI9VpnBlQ+e7VWfOQaO7/e6KpEapSYnMq5/Fu+u3cX2A8e9LieqKNBjXZ9JMG4W7FwN00fBES0mIJFvyqAcTjvHi59t8bqUqKJAjwddb4AJc6B4PTw/Ag5oHUeJbFktG3F117a8tGwLJ0rUwhgoBXq86Hw1TFngP0N/9jrYtdbrikQuKj8vh+Kjp3hztcZaBEqBHk9y8uCOtwEHzw2HjR97XZHIBQ3p3JpL0hvr5mgtKNDjTbsr/A8gNesAs2+BNfO9rkikWmZGfp6P1UUHWblFN/QDoUCPR82z4M53ICMXXrkLFj/udUUi1bq5byZNGiZpCmOAFOjxKrUFTHkVuo+G934O7zysp0ol4jRpmMTYfpm8uWYHew6f9LqciBdQoJvZcDP7xswKzeyhC2wzzMxWmdlaM/tLcMuUkGiQAmOnw8C/haVPwvw7oOSE11WJnGPq4BxKyhwvLVMLY01qDHQzSwSeAEYA3YEJZta9yjbNgSeBHzrnLgduDX6pEhIJCTD8t3Ddv8JXr8Hsm/UAkkSUTulNuLJLOi98tpmSMv0r8mICOUMfABQ65zY4504Bc4DRVbaZCCxwzm0BcM7p6ZVoYgZ5P4FbnoWty/wdMAe1YK9EjvzBOew6dJJ31+70upSIFkigZwCVn0QpKn+tsi5ACzP7yMxWmNnU6j7IzO4xswIzK9izZ0/dKpbQ6TEWJr8Ch7bDM9fAzi+9rkgEgGGXtSG7ZSPdHK1BIIFu1bxWda5lEtAPuAG4HvilmXU574ece9o5l+ucy01PT691sRIGnb5f3quO/6lS9apLBEhMMKYOzmH5pv2s3X7Q63IiViCBXgRkVfo+E9hezTbvOOeOOuf2Ah8DvYJTooRd5V71WTerV10iwq39skhtkKiz9IsIJNCXA5eaWUczSwbGAwurbPM6MNTMksysETAQWBfcUiWsKnrVswaU96r/J2jBAfFQWqMGjOmTweurtrP/qGb8V6fGQHfOlQL3Ae/iD+l5zrm1ZjbNzKaVb7MOeAdYDSwDnnHO6QJstEttAZMXQPcx8N4v1KsunsvPy+Fk6WnmFmjAXHXMq2WecnNzXUFBgSf7llo6fdr/8NHSJ/3hftN/+XvYRTww7r+WULT/OB//01UkJlR3iy+2mdkK51xude/pSVGpmXrVJYLcnudj24HjfLhul9elRBwFugSuaq+65qqLB67t3pb2aSmawlgNBbrUTo+x/rnqh7bDs9eqV13CLikxgcmDcvi0sJjC3VortzIFutRexyv9HTCYetXFE+P7Z5GcmMCMxZu9LiWiKNClbtpeDne/D80yNFddwq5Vk4aM6tWeBZ8XcfhEidflRAwFutRdWibc+TZk9levuoTd7Xk+jp4q45UVmjtUQYEu9aNedfFIz8zm9Mluzswlmzl9WicSoECXYGiQAmOfh0H3wmd/1Fx1CZv8wT427D3KJ4V7vS4lIijQJTjUqy4eGNmjPa2bNNR8l3IKdAku9apLGCUnJTBxYDZ//mY3m4uPel2O5xToEnzqVZcwmjQwm0QzZi1RC6MCXUKj45Xlc9XLe9U3aJlZCY22zVIYfkU75hZs5dipUq/L8ZQCXUKn3RXqVZewuD3Px+ETpby6cpvXpXhKgS6hVdGrnjXQ36v+6WPqVZeg65fTgss7NGPm4s14NUE2EijQJfRSW/jXKu0+Bt7/pXrVJejMjPzBPr7ZdZilG/Z5XY5nFOgSHlV71V+5E0q16owEzw97d6BFowZx3cKoQJfwqehVv/Z/wdpXYe5kKDnudVUSI1IaJDKufzbvfbWTbQfi88+VAl3C73t/B6P+AN+9By/cCiePeF2RxIjJg7IBmL00PlsYFejijdw7/UvZbV4MM0fD0WKvK5IYkNmiEdd0a8ucZVs4WVrmdTlhp0AX7/QaB7fNhF1f+h9A2rfR64okBkwelMP+YyW8uzb+lqhToIu3uo2Cqa/DsWJ/qG9f6XVFEuWGdG5NZotU5izb4nUpYadAF+9lD4K73oOkVHj+BvjuA68rkiiWkGCMy81i8fpiNu2Nr/kuCnSJDOmX+Z8qbdUJXhoHq170uiKJYrfmZpGYYMxZHl/D4RToEjmatoPb3wLfEHjtb+Hj/6OnSqVO2qWlcNVlbZi/YiunSuPnITYFukSWlGYw8WXoOQ7+9K/w5gNQFt8Dl6RuJg7MYu+RU3y4Ln5ujirQJfIkJcOYp2DIT6HgOf8DSKeOeV2VRJnvd2lD+7QUXoqjyy4KdIlMCQlwzSMw8lH49h2YcSMc1TJjErjEBOO23Cw++W5P3NwcVaBLZBvwIxg3q7xX/Tr1qkutTCxf/CJenhxVoEvk63ajetWlTioWv5gXJ4tfKNAlOqhXXero9jwfh+Jk8QsFukSPqr3qK1/wuiKJAvG0+EVAgW5mw83sGzMrNLOHLrJdfzMrM7OxwStRpJLKveqv36tedalRPC1+UWOgm1ki8AQwAugOTDCz7hfY7vfAu8EuUuQcVXvVF/1UvepyUfGy+EUgZ+gDgELn3Abn3ClgDjC6mu1+ArwC7A5ifSLVS0r2j98d8lNY8TzMm6JedbmgeFn8IpBAzwAqd+YXlb92hpllADcBT13sg8zsHjMrMLOCPXv21LZWkXOZne1V/+ZtmPlDzVWXC6pY/OKFGG5hDCTQrZrXql60/A/gZ865i06Ud8497ZzLdc7lpqenB1iiSA0qetV3rtFcdbmgM4tfLN/KiZLYXPwikEAvArIqfZ8JbK+yTS4wx8w2AWOBJ81sTDAKFAnIOb3q18H2VV5XJBHo9jwf+46eYtHqHV6XEhKBBPpy4FIz62hmycB4YGHlDZxzHZ1zPuecD5gP3Oucey3YxYpc1Jle9RR4fqR61eU8gy9pxaVtmjBj8aaYbGGsMdCdc6XAffi7V9YB85xza81smplNC3WBIrWiXnW5CDNjap6PNdsOsnLrAa/LCbqA+tCdc28557o45y5xzv1b+WtPOefOuwnqnLvdOTc/2IWKBKxqr/pf1KsuZ93cJ4OmDZNisoVRT4pKbKrcq/5n9arLWY0bJjE2N5O31uxg9+ETXpcTVAp0iV1netUf8Peqa666lMsf7KOkzPHSZ7E1K12BLrHNDK751dm56upVF8DXujHDLkvnhc82x9QSdQp0iQ/qVZcq8gf72H34JO+s3el1KUGjQJf4obnqUsn3u6Tja9WImTF0c1SBLvFFc9WlXEKCMWWwj4LN+/ly20GvywkKBbrEH/WqS7mx/TJJbZAYMy2MCnSJT2d61YdqrnocS0ttwM19M3j9i+3sO3rK63LqTYEu8SulGUycp7nqcS4/z8ep0tPMXR79LYwKdIlvVeeqq1c97nRp25TBnVoxe+lmSsuiu4VRgS5Sea76t+/AjBvVqx5n8vN8bDtwnA/WRff6PAp0kQoDfgTjZsOuL2H6SDgcO/3JcnHXdGtDRvNUZi7Z5HUp9aJAF6ms2yiY9DIc2ArPDYcDW7yuSMIgKTGByYNyWLy+mG93Hfa6nDpToItU1fFKmPoaHNsHz42Avd95XZGEwbj+WSQnJUR1C6MCXaQ6WQPg9kVQesK/AtLW5V5XJCHWsnEyo3t1YMHn2zh4vMTrcupEgS5yIe17+p8qTUnz3yj9+i2vK5IQy8/zcbykjPkrirwupU4U6CIX0+oSuOt9aNMN5k6Cgue8rkhC6IqMNHJzWjBrySZOn46+B80U6CI1aZLuv/zS+Rr/w0d/+lc9VRrDpub52FR8jL98t8frUmpNgS4SiOTGMP4l6DvVPybgtXuhLDqvs8rFjbiiHW2aNozKm6MKdJFAJSbBjY/BsIfhixfhxXFwMnpb3KR6DRITmDQwh4++2cPGvUe9LqdWFOgitWEGwx6CHz4OGz6C6TfA4V1eVyVBNmFgFg0SLeoeNFKgi9RF3ykwYY6/R/3Za2FvodcVSRC1aZrCyB7tmV9QxNGT0TOwTYEuUlddrvPfLD111B/q6lWPKVMH+zh8spQFK7d5XUrAFOgi9ZHRT73qMapvdnN6ZKQxc/EmXJR0NSnQReqraq/68me9rkiCwMzIz/Px3e4jLFkfHdM3FegiwXCmV/1aePMB9arHiFE929OycTLTo6SFUYEuEizJjWH8i2d71V//sXrVo1xKg0TG98/ig3W7KNof+QufKNBFgqlyr/qqF8p71Y94XZXUw6RBOQDMWrrZ40pqpkAXCbYzver/Wd6rPlK96lEso3kq13Vvx9zlWzlRUuZ1ORelQBcJlb5TYcJLlXrVNVc9WuXn+ThwrISFq7Z7XcpFBRToZjbczL4xs0Ize6ia9yeZ2eryr8Vm1iv4pYpEoS7XV+pVvw62LvO6IqmDQZ1a0rVdU6ZHeAtjjYFuZonAE8AIoDswwcy6V9lsI/B951xP4DfA08EuVCRqZfSDu9+v1Kv+ptcVSS2ZGVMH+/hqxyFWbN7vdTkXFMgZ+gCg0Dm3wTl3CpgDjK68gXNusXOu4iiXApnBLVMkyrXsVN6r3h3mTlavehQa06cDzVKSIrqFMZBAzwC2Vvq+qPy1C7kLeLu6N8zsHjMrMLOCPXuib9awSL1U7VX/8DfqVY8ijZKTuC03i3e+3MmuQye8LqdagQS6VfNatX8Kzewq/IH+s+red8497ZzLdc7lpqenB16lSKyo3Kv+yaOaqx5lpgzOocw5Xvhsi9elVCuQQC8Csip9nwmcd6vXzHoCzwCjnXPR8ZysiBc0Vz1q5bRqzFWXteHFz7ZwqvS01+WcJ5BAXw5camYdzSwZGA8srLyBmWUDC4Apzrlvg1+mSIw5r1ddc9WjRX6ej71HTvLWmh1el3KeGgPdOVcK3Ae8C6wD5jnn1prZNDObVr7ZvwCtgCfNbJWZFYSsYpFY0neq5qpHmaGdW9OpdeOIvDkaUB+6c+4t51wX59wlzrl/K3/tKefcU+W/vts518I517v8KzeURYvEFM1VjyoJCcaUwTms2nqAL7Ye8Lqcc+hJUZFIUNGrntpcc9WjwNh+mTROTmRGhC1Rp0AXiRQVveptu2uueoRrmtKAW/plsuiLHew9ctLrcs5QoItEksatIf8N9apHgamDczhVdpq5y7fWvHGYKNBFIs2ZXvV89apHsM5tmjKkc2tmLdlMaVlktDAq0EUiUWIS3Pj/Ydg/n+1VPxX5CyzEm/w8HzsPneC9ryKj5VSBLhKpzGDYz/wPIW34M8y+BU4c8roqqeQHXduQ2SKVGRHSwqhAF4l0/fLhlmegaBnMHA3H9nldkZRLTDCmDMrhs437WLfD+79sFegi0eCKW2DcbNi11v9U6cFtXlck5cb1z6JhUgIzl3i/RJ0CXSRaXDYCJs2DA1v9DyDtXud1RQI0b5TMmN4ZvLZyGwePeXvzWoEuEk06DYM73oLTpfDc9bDpU68rEvw3R4+XlDGvwNsWRgW6SLRp39P/AFLjNjBrDKx91euK4l73Ds0Y4GvJzKWbKDvt3XMDCnSRaNQiB+56Dzr0hZfvgCVPel1R3Jual8PWfcf56JvdntWgQBeJVo1awtTXoOsN8O7D8O7P4XRkPOASj66/vB3tmqUww8Obowp0kWjWIBVumwkD7oElj8OCu6E0cmaLxJMGiQlMGpjNx9/uYf2eI57UoEAXiXYJiTDi3+GaX8OXr5Q/gHTQ66ri0vgB2SQnJjDLo7N0BbpILDCDIX8PNz0NW5bAcyPg0HkrRUqIpTdtyA092zN/RRFHTpaGff8KdJFY0mscTJoPB7bAM+pV90J+no8jJ0tZ8HlR2PetQBeJNZdcVaVX/a9eVxRXemc1p1dmGjMWb8KFefSxAl0kFrXv6V8BqUlbmHWTetXDLD/Px/o9R/lr4d6w7leBLhKrmmfDne+e7VVf+kevK4obN/RsT+smycxYHN6bowp0kVhW0avebRS885B61cOkYVIiEwZk8+HXu9i6L3xz7BXoIrGuQSrcOkO96mE2cWA2CWbMWhq+s3QFukg8ONOr/oh61cOkfVoqwy9vx9zlWzl+qiws+1Sgi8QLMxjy0/Je9aX+XnXNVQ+pqYNzOHi8hNdXhee/swJdJN70GgeTXvb3qmuuekgN6NiSru2aMj1MLYwKdJF4pF71sDAzbs/z8fXOwyzbGPqlAxXoIvGqYq56Ra/6lwu8rigmje6dQVpqg7AsUadAF4lnLXLO9qrPv1Nz1UMgNTmRcf2zeGftTnYcPB7SfSnQReKd5qqH3JRBOZx2jheWbgnpfhToIqK56iGW1bIRV3dtw0vLtnCiJHQtjAp0EfGrbq768QNeVxUz8vN8FB89xVtrdoRsHwEFupkNN7NvzKzQzB6q5n0zs8fK319tZn2DX6qIhFzFXPWb/9vfq/78SLU1BsmQzq3plN6YGYs3hWwfNQa6mSUCTwAjgO7ABDPrXmWzEcCl5V/3AJoCJBLNet4Gk+fDoW3wx+/Bmw/C0WKvq4pqZkb+YB9fFB1k5Zb9IdlHUgDbDAAKnXMbyouaA4wGvqq0zWhgpvN3zi81s+Zm1t45F7p/W4hIaHUaBvevhD//byh4Dr6YA2kZXlcV1SY7x/caHuXbd8bQ555/C/rnBxLoGcDWSt8XAQMD2CYDOCfQzewe/GfwZGdn17ZWEQm3Ri3hhkeh/13+m6UnD3tdUVRLBNKbluDr1SMknx9IoFs1r1V9hjWQbXDOPQ08DZCbmxvepTxEpO7adIPRT3hdRUxIC+FnB3JTtAjIqvR9JlB19dlAthERkRAKJNCXA5eaWUczSwbGAwurbLMQmFre7TIIOKjr5yIi4VXjJRfnXKmZ3Qe8i/8S0HPOubVmNq38/aeAt4CRQCFwDLgjdCWLiEh1ArmGjnPuLfyhXfm1pyr92gE/Dm5pIiJSG3pSVEQkRijQRURihAJdRCRGKNBFRGKEhWOdu2p3bLYHqOsSHq2BvUEsJxromOODjjk+1OeYc5xz6dW94Vmg14eZFTjncr2uI5x0zPFBxxwfQnXMuuQiIhIjFOgiIjEiWgP9aa8L8ICOOT7omONDSI45Kq+hi4jI+aL1DF1ERKpQoIuIxIiIDvR4XJw6gGOeVH6sq81ssZn18qLOYKrpmCtt19/MysxsbDjrC4VAjtnMhpnZKjNba2Z/CXeNwRbAn+00M3vDzL4oP+aontpqZs+Z2W4z+/IC7wc/v5xzEfmFf1TveqATkAx8AXSvss1I4G38KyYNAj7zuu4wHHMe0KL81yPi4Zgrbfcn/FM/x3pddxh+n5vjX7c3u/z7Nl7XHYZj/mfg9+W/Tgf2Acle116PY74S6At8eYH3g55fkXyGfmZxaufcKaBicerKzixO7ZxbCjQ3s/bhLjSIajxm59xi51zFkuFL8a8OFc0C+X0G+AnwCrA7nMWFSCDHPBFY4JzbAuCci/bjDuSYHdDUzAxogj/QS8NbZvA45z7GfwwXEvT8iuRAv9DC07XdJprU9njuwv83fDSr8ZjNLAO4CXiK2BDI73MXoIWZfWRmK8xsatiqC41AjvlxoBv+5SvXAH/nnDsdnvI8EfT8CmiBC48EbXHqKBLw8ZjZVfgDfUhIKwq9QI75P4CfOefK/CdvUS+QY04C+gFXA6nAEjNb6pz7NtTFhUggx3w9sAr4AXAJ8L6ZfeKcOxTi2rwS9PyK5ECPx8WpAzoeM+sJPAOMcM4Vh6m2UAnkmHOBOeVh3hoYaWalzrnXwlJh8AX6Z3uvc+4ocNTMPgZ6AdEa6IEc8x3A75z/AnOhmW0EugLLwlNi2AU9vyL5kks8Lk5d4zGbWTawAJgSxWdrldV4zM65js45n3POB8wH7o3iMIfA/my/Dgw1syQzawQMBNaFuc5gCuSYt+D/Fwlm1ha4DNgQ1irDK+j5FbFn6C4OF6cO8Jj/BWgFPFl+xlrqonhSXYDHHFMCOWbn3DozewdYDZwGnnHOVdv+Fg0C/H3+DTDdzNbgvxzxM+dc1I7VNbOXgGFAazMrAn4FNIDQ5Zce/RcRiRGRfMlFRERqQYEuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLiISIxToIiIx4n8AV94s01x57vIAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(df_ideal[\"threshold\"], df_ideal[\"tpr\"], label=\"TPR\")\n",
"plt.plot(df_ideal[\"threshold\"], df_ideal[\"fpr\"], label=\"FPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "e40c7294-24b6-4b11-804d-0bdf95e35581",
"metadata": {},
"source": [
"* cut at threshold 0.726\n",
"* TPR is maximized and FPR is minimized for this threshold\n",
"* Such an ideal moel does not exist in reality, but it helps us to see how good our model is"
]
},
{
"cell_type": "markdown",
"id": "c4363131-ecaf-44c7-aea2-5cc665c13dbc",
"metadata": {},
"source": [
"### Putting everything together"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "9e469e37-84ff-4cf1-84b7-63b5558af2f4",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABK50lEQVR4nO3dd1hU19bA4d9mAGkKCnZAsMUuCBawYu81dmMvqJCYmNyYdmO+xOSmXJMY7L3EGjv2XjGKvcXesIsVFWn7+2PESwwiwsycmWG/z8OjUzh7nUgWe/ZZZ20hpURRFEWxfDZaB6AoiqIYhkroiqIoVkIldEVRFCuhErqiKIqVUAldURTFSthqNbCHh4f08fHRanhFURSLtH///jtSyvzpvaZZQvfx8SE6Olqr4RVFUSySEOLSq15TSy6KoihWQiV0RVEUK6ESuqIoipXQbA1dURTFEBITE4mJiSE+Pl7rUAzKwcEBT09P7OzsMv09KqErimLRYmJiyJ07Nz4+PgghtA7HIKSUxMbGEhMTg6+vb6a/77VLLkKIaUKIW0KIY694XQghxgghzgohjgghqrxB3IqiKNkSHx+Pu7u71SRzACEE7u7ub/ypIzNr6DOAphm83gwo9fxrIDD+jSJQFEXJJmtK5qmyck6vXXKRUm4XQvhk8JY2wCyp78O7RwjhJoQoLKW8/sbRZMIfsyfx5b//TYW6rfHIlxc3J3t0Ntb3j6ko5sDR0ZH+/fuTP3+697EoZsYQa+hFgStpHsc8f+4fCV0IMRD9LB5vb+8sDbZ4yRJOXLzJiYuTU48K4sXfFEUxICklkydPJjIyknLlymkdjlmKjY2lQYMGANy4cQOdTvfiF+Dhw4epXLkySUlJlC1blpkzZ+Lk5IROp6NixYokJSXh6+vL7NmzcXNzy3Yshkjo6eXRdHfNkFJOAiYBBAYGZmlnjXmTfqUZfvRZ8YzSlQLp8vlYrsQJztx6xOW7T0jdr8Pe1oY6pfLTslJhGpQtQG6HzF8pVhRFb+/evbRu3ZqgoCAWLlxIkyZNtA7J7Li7u3Po0CEARo4ciYuLCx9++CEALi4uL17r3r07EyZM4IMPPsDR0fHF87169WLs2LF89tln2Y7FEAk9BvBK89gTuGaA46Yvrw89/RxwqtyObt/+wYpvQ1mzZg0FChTgaUIy527HcfrmI47EPGDd8RtsPHkTe1sb6pXOT4tKhWlQtiAuuVRxj6JkRrVq1di3bx+tWrWiefPmbNy4kZCQEK3Dski1a9fmyJEj/3g+KCgo3eezwhCZbQUQJoSYD1QHHhhr/RwA21zg6snbno44LVtGhw4dCA4OZu3atZQsWZIKRV2pUNSV9lU8+XfLchy8co+Vh6+z5th11p+4SS5bG0LeKvA8uRfAyV4ld0XJiJeXFzt37qR06dL8+OOPZp3Qv1p5nBPXHhr0mOWK5OHLVuWzdYykpCTWrFlD06Z/ry9JTk5m06ZN9OvXL1vHT5WZssV5QBTwlhAiRgjRTwgRKoQIff6W1cB54CwwGRhikMgykq8E3D1H8+bN2bx5M/fv3yc4OJh9+/b97W02NoKAYvkY2bo8USMasCg0iC5Vvdh/+R7h8w5S5esNTN15AbWvqqJkzMXFhdBQ/afhM2fOaB2OxXj69Cl+fn4EBgbi7e39InGnPu/u7s7du3dp1KiRYQaUUmryFRAQILMs8gMpv/WSMiVFSinlqVOnpK+vr3RycpKrVq167bcnJafIqHN3ZJ/pe2WxjyPliMVHZEJSctbjUZQc4Pr169LOzk6+9957WofyNydOnNA6hBe+/PJL+eOPP7547OzsnO77Up+/f/++rFWrlvz111/TfV965wZEy1fkVcvs5ZKvBDx7AE9iAShdujS7d++mTJkytG7dmqlTp2b47TobQY3i7kzpGciQeiWYt/cyvabt5cDlezyKTzTFGSiKxSlUqBAdO3Zk+vTpxMXFaR2OVXB1dWXMmDH89NNPJCZmP/dYZkJ3L6H/M/bci6cKFSrE1q1badiwIf379+err7567VKKjY3gX03L8FPHykRfvEf7cbupOHI9wd9t4sNFh9ly6hYJSSnGPBNFsShhYWE8fPiQ2bNnax2K1fD396dy5crMnz8/28cSr0t6xhIYGCizvMHFnbMQEQBtx4Nft7+9lJiYyIABA5g5cyb9+/dn/Pjx2Nq+/sLnzYfxHIl5wOmbjzh5/SHbTt/mUXwSro52dK7qxcA6xfFwyZW1eBXFSkgpqVq1Kk+fPuXYsWNmcYfmyZMnKVu2rNZhGEV65yaE2C+lDEzv/ZZZ4pG3GAjd32boqezs7Jg+fTqenp6MGjWK69evs2DBApydnTM8ZME8DjQq50CjcgUBeJaUzI7Td1h66CpTdpxndtQlegYXo0MVT3zcnbG3tcwPN4qSHUIIwsPD6d27N1u2bKF+/fpah6SkYZkzdIBf/aCIH3Sc8cq3TJw4kSFDhhAQEEBkZCQFChTI0lDnbscxZtMZVhy+hpRgayPw8XAmn7P9i/fULZ2fwXVLYKPaEChWLj4+Hi8vL2rVqsXSpUu1DkfN0NOw3Gmme4l0Z+hpDRo0iKVLl3Ls2DGCg4M5e/ZsloYqkd+FX7v4s+3DEH7p7MfAOsXxcXfGRoCNgPjEZH5cd4qweQd4mpCcpTEUxVI4ODgwYMAAVqxYwaVLr9zeUtGA5Sb0fCXg7nl4zSeM1q1bs2nTplfWqr8Jb3cn2voX5V9NyzClVyDzBwYxf2AQy4fW5PMWZVlz7AadJkZx86F1NdpXlJcNHjwYgPHjVXNVc2K5Cd29BCTEQdyt1741KCiI3bt34+LiQr169Vi9erVBQxFC0L92cSa/E8i523G0idjFsasPDDqGopgTLy8v2rZty+TJk3n69KnW4SjPWW5Cz/e8dPFuxssuqd60Vj0rGpYryB+hwdgI6DghirXHbhh8DEUxF+Hh4dy9e5d58+ZpHYrynOUmdPfi+j9fs46eVlZq1d9UuSJ5WBZWk7cK5SZ0zn7GbDpDUrKqZVesT926dalQoQK//fZbjm+fodPp8PPze/F18eJFtm7diqurK/7+/pQtW5avvvoK4G/PlylT5kVnRkOw3ITu6g02tpmeoafKnTs3K1eupFevXowcOZKBAweSlJRk0NAK5HZg/sAatPUrwugNp3l7QhRnb6k76xTrIoQgLCyMQ4cOsWvXLq3D0VRqO9zULx8fH0DfYfHgwYNER0czZ84c9u/f/7fnDx48SGRkpMH++1luQtfZQl6fN5qhp0qtVf/ss8+YMmUKbdu25fHjxwYNz8FOx8+d/fitqz8XYx/TfMwOJm8/T3JKzp7JKNalR48e5MmThwkTJmgdillzdnYmICCAc+f+nq8cHR3x8/Pj6tWrBhnHMm8sSpVa6ZIFQgi++eYbvLy8GDJkCCEhIdmqVX/VGK0qF6F68Xx8tvQYo1afZO3xG/zUsTK+Hhnf6KQolsDZ2ZkePXowdepUxowZQ758+bQNaM0IuHHUsMcsVBGa/SfDt6R2TwTw9fX9R31+bGwse/bs4YsvvuD27dsvnr937x5nzpyhTp06BgnVcmfooK90yUTpYkYMVauekQK5HZj0TgA/d67MmZuPaPbrdn5Y+xdrj93g/O04NWtXLNqAAQN49uxZju7vknbJJW0y37FjB/7+/jRu3JgRI0ZQvnz5F89XqlSJQoUK0bJlSwoVKmSQOCx8hl4cEp/Ao+uQp0iWD9O6dWs2b95My5YtCQ4OJjIykmrVqhkwUP1svZ2/J8ElPPhs6VHGbf3fRy9XRzualC9Ii0pFCC7hjp3Osn/PKjmLn58fVatWZfLkybz77rva9nd5zUza1GrXrk1kZOQrnz99+jS1atWiXbt2L2b42WHZmcOjlP7P26eyfagaNWq8qFUPCQlh1apV2T5megrmcWBKr6oc+6oJy4bW5Ie3K9GgTAFWH71Br2l7qf/frVyOfWKUsRXFWAYMGMDx48eJiorSOhSLUrp0aT755BO+//57gxzPshN6wQr6P28eM8jh0taqt2nThilTphjkuOlxyWWLn5cbnQK9GN3Zj+jPGzKhRxUexSfRdfIertxVSV2xHF26dMHFxYXJkydrHYrFCQ0NZfv27Vy4cCHbx7LshO7sAbmLGPQiSNpa9QEDBjBy5EiT1Ng62OloWqEwc/pVJ+5ZEl0m7SHmnkrqimXInTs3Xbt2ZcGCBTx4kPPukk5vw4969eqlu9zy8vOOjo5cvXoVX1/fbMdh2Qkd9FegDXxVO7VWvXfv3nz11VcMGDDA4LXqr1KhqCtz+lXnUXwinSfu4czNRyYZV1Gya8CAATx9+pQ5c+ZoHUqOZR0J/fYpSDRsQyw7OzumTZvG559/ztSpU41Sq/4qFT1d+b1/DZ4lpdB+3G62nb79+m9SFI0FBgYSEBDA2LFjc/ydo1qxjoQuk+H2SYMfWgjB119/zYQJE1izZg0hISHcuvX6ZmCGUNHTleVhNSma15E+0/cyY9cF9T+JYtZS7xw9efIkmzdv1jqcHMk6EjoY/maCNExRq56eom6O/DE4mPplCjBy5Qn6ztinWvMqZq1Lly54eHjw22+/aR1KjmT5CT2vL9i7GDWhw/9q1Q3RV/1NuOSyZdI7gXzZqhxR52NpNHobyw4a5jZhRTG01M0vVq5cycWLF7UOJ8ex/IRuY6MvXzRyQoe/16rXq1fPaLXqL7OxEfSp6cua9+pQqmBuhi04xMRtb97DRlFMITQ0FFCbX2jB8hM6PK90OQYpxm9TW7p0aaKiokxSq/4yXw9nFgysQavKRfhuzV9M3p61PjaKYkze3t60bduWKVOm5IjNL2JjY1+0zS1UqBBFixZ98VgIgZ+fHxUqVKBjx448eaIvRU5tt1uhQgVatWrF/fv3DRKL9ST0hEdw3zT7GxYsWPBvterG6Kv+KrY6G37uVJkWFQszavVJpuw4ry6WKmYnJ21+4e7u/qKPS2hoKO+///6Lx87Ozhw6dIhjx45hb2//oitlau+XY8eOkS9fPsaOHWuQWKwnoYNJll1SvdxXPTQ0lORk02wQbauz4ZcufjSrUIhvVp3k7QlR7DxzRyV2xWyozS/+qXbt2ukWVAQFBan2uX9ToCwInT6hl2ttsmFT+6oXKVKE7777jri4OGbMmIGdnZ3xx9bZ8FtXf+btu8K4LWfpMfVPqvnm45u2FShdMLfRx1eUjAghCA8PZ9CgQezatYtatWqZZNxhw4Zx6NAhgx7Tz8+PX375JVvHSEpKYs2aNTRt2vRvzycnJ7Np0yb69euXreOnso4Zup0jeJQ26Qw9lRCCb7/9lm+//Za5c+fSqVMnnj17ZpKxbXU2vFOjGFs/qsf/tSnP2VtxtByzk/Fbz6mWvIrmunfvjpubGxEREVqHopnUPumBgYF4e3u/SNypz7u7u3P37l0aNWpkkPGsY4YO+mWXS7s1G/6TTz7BxcWFd999l5YtW7J48WLy5MljkrFz2eroGeRD84qF+XzpMb5f+xfrT9xgSs9A3F1ymSQGRXmZs7Mzffv2ZcyYMVy7do0iRbLe4jqzsjuTNrTUtfJXPf/gwQNatmzJ2LFjeffdd7M9nnXM0EGf0B/GwJO7moUQHh7OjBkz2LJlC3Xr1uX69esmHd/DJRfje1Th1y5+nLj2kIGz9xOfaJp1fUVJz5AhQ0hOTlZb1L2Cq6srY8aM4aeffiIxMTHbx8tUQhdCNBVCnBJCnBVCjEjndVchxEohxGEhxHEhRJ9sR/amClfS/3ntoMmHTqtXr15ERkZy5swZgoKCOHnS8C0JMiKEoI1fUUZ38mP/pXt8vPiIuiilaKZEiRI0b96ciRMnmmwp0tL4+/tTuXJl5s+fn/2DSSkz/AJ0wDmgOGAPHAbKvfSeT4Hvn/89P3AXsM/ouAEBAdKg4h9KOTKvlBv/z7DHzaLo6GhZsGBBmTdvXrlz505NYojYfEYW+zhS/rzhlCbjK4qUUq5du1YCcs6cOUY5/okTJ4xyXHOQ3rkB0fIVeTUzM/RqwFkp5XkpZQIwH2jz8u8FILfQ7z3l8jyhm6bfbKpcuaGIH1zaZdJhXyUgIIDdu3fj4eFBw4YN/7FprCkMqVeCDlU8+WXjGcZuOatm6oomGjVqROnSpXP0xVFTyUxCLwpcSfM45vlzaUUAZYFrwFHgPSnlP27bFEIMFEJECyGi0+58bTDFasLV/ZBoHnenFS9enN27d1O5cmU6dOhgsJsHMksIwXftK9K6chF+XHeK4YsO8yxJrakrpmVjY8PQoUPZs2cP0dHRWodj1TKT0NPb8fXlqV4T4BBQBPADIoQQ/yjxkFJOklIGSikD8+fP/4ahZkKxmpCcADGmaZyVGR4eHmzevJlWrVoRFhbGJ598YtKZsr2tDb928eODRqVZcuAq3Sb/yeL9MRyNecDTBJXcFdPo3bs3Li4uqgujkWUmoccAXmkee6KfiafVB1jyfInnLHABKGOYEN+Adw1AwEXzWHZJ5eTkxOLFixk0aBD/+c9/6NWrFwkJCSYbXwjBuw1KEdHNn1M3HjF80WFaReyk0lfrWHIgxmRxKDlXnjx56NWrF/Pnz8con84VIHMJfR9QSgjhK4SwB7oAK156z2WgAYAQoiDwFmD6zlGObs/r0c0roQPY2toyfvx4vvnmG2bPnk2LFi14+PChSWNoWakIh/7diI0f1GV89yr4e+fl48VH2HM+1qRxKDnT0KFDSUhIUBtJG9FrE7qUMgkIA9YBJ4GFUsrjQohQIUTo87d9DQQLIY4Cm4CPpZR3jBV0hnxq6ZdcksyvREoIwWeffcb06dM1q1W31dlQsoALzSoWZnLPQLzzORE6Zz8X7phmez0l5ypbtiwNGzZk/PjxJtujN6fJVB26lHK1lLK0lLKElHLU8+cmSCknPP/7NSllYyllRSllBSmldrvEFqsJSfH6i6Nmqnfv3prWqqdydbRjWu+qCKDfjH3cfWy6ZSAlZwoPDycmJobly5drHYpBpbbDTf26ePEiW7duxdXVFX9/f8qWLctXX30F8Lfny5Qpw4cffmiwOKznTtFUxYL1f5rZOvrLmjZtyrZt23j69Ck1a9Zk1y5t4i3m7syknoHE3H9K+3G7OHc7TpM4lJyhRYsW+Pj4WN3F0dRb+VO/fHx8AH2HxYMHDxIdHc2cOXPYv3//354/ePAgkZGRBvv/3/oSulM+KFAeLu3UOpLXCggIICoqStNadYCqPvmY2786j+KTaDd2F7vOarNaplg/nU7HkCFD2LZtG0ePmr6ZnlacnZ0JCAjg3Lm/7zTm6OiIn5+fap+bIZ+acHAOJCeCzvitbLMjtVa9ZcuWdOjQgd9++42hQ4eaPI5An3wsG1qTfjP30XPaXn7r6k/zioVNHodi/fr168e///1vIiIimDhxokGPrVX73NTuiQC+vr7/mJzFxsayZ88evvjii79V+dy7d48zZ85Qp04dg8RqfTN00K+jJz6Bqwe0jiRTXq5VHzFihCZ3dXrlc2Lx4GD8vdx4f8EhDl6+Z/IYFOuXL18+unfvzpw5c7h3zzp+xtIuuaRN5jt27MDf35/GjRszYsQIypcv/+L5SpUqUahQIVq2bEmhQoUME8iregIY+8vgvVzSehz7vK/LV8YbwwgSExPloEGDJCDfeecd+ezZM03iiI17Jmt/v1kGfL1eXo59rEkMinU7dOiQBOR///vfbB/LHHq5ODs7/+O5LVu2yBYtWmT4/KlTp2T+/PnlwYMH0z2uMXq5WB6nfOAdBH+t1jqSN2IOteoA+Zztmda7KglJKfSbuY+H8dlv66koaVWuXJnatWszduxYk23daI5Kly7NJ598wvfff2+Q41lnQgco0xxun4S7pr+/KTvS1qpv3bpVk1p1gJIFXBjfI4Dztx/TcsxO/lQ3HykGFhYWxvnz51mzZo3WoWgqNDSU7du3c+HChWwfS0iNOvAFBgZKozbquXsBxvhB41EQHGa8cYxo3bp1dOjQAQ8PD9asWUPZsmVNHsPeC3f5cNFhrtx7Qu9gHz5oVJrcDuZ9oVmxDImJifj6+lK+fHnWrVuX5eOcPHlSk/83TCG9cxNC7JdSBqb3fuudoefz1ZcvnrKsZZe0mjRpwrZt24iPj9esVr2abz7WDqtNzxrFmL7rIgHfbGTgrGiWH7pK3DN1t5+SdXZ2doSGhrJ+/XpOnTqldThWwXoTOuiXXS5HwWPLXS5IrVXPnz+/ZrXqTva2fNWmAivCatK9ujeHY+7z3vxDBHy9gdDZ+1l5+BpPElRyV97cgAEDsLe3N3lraWtl3Qn9reYgU+BM1j/OmQNfX1927dqFn5+fJn3VU1XydOPLVuWJGtGAhYOC6FrNm/2X7xE+7yBVvt7AkN/3s+rIddWWV8m0ggUL0qlTJ2bMmMGjR4+yfBytlo6NKSvnZN0JvYg/5C4Cf63SOpJs8/DwYNOmTZr1VU/LxkZQzTcfI1uXZ88nDZg/sAYdA7zYe+EuQ+ceoMrXGwibe4C1x66rTaqV1woPD+fRo0fMmjUrS9/v4OBAbGysVSV1KSWxsbE4ODi80fdZ70XRVJEfwOF58K/zYOdo/PGMLCkpibCwMCZOnEiPHj2YOnUq9vb2WocFQHKK5M8Lsaw6cp21x24Q+ziBUgVcmNqrKt7uTlqHp5ixatWq8ejRI06cOIF+J8vMS0xMJCYmhvj4eCNFpw0HBwc8PT2xs/t7EUJGF0Wt88aitM5skPLLPFIeX2aa8UwgJSVFfv311xKQDRs2lA8ePNA6pH9ITEqWa49dl5VGrpP+/7de7r0Qq3VIihmbNWuWBOT69eu1DsXskeNuLErLty64FYOdP4OVfCQTQvD555+/6Ktep04drl17eRMpbdnqbGhSvhDLhtbEzdGO7pP/ZNzWs1x/YB77vSrmpVOnThQoUMDqujCamvUndJ0d1B4O1w7C2Y1aR2NQqX3Vz549q2lf9Yz4ejizdEhNgku688PaUwR9t5m3x+8m8oh5/QJStJUrVy4GDhxIZGSkQW6wyamsP6EDVO4Krl6w9T9WM0tPldpXPbVWfedO82sb7Opkx4w+1dg8vC7DG5Xm/tNEwuYe5LvVJ0lOsa5/DyXrQkNDsbGxYdy4cVqHYrFyRkK3tYfaH8DVaDi/RetoDC5tX/VGjRpp1lf9dYrndyG8QSnWvlebd2oUY+L284TO2c9jdYOSAhQtWpT27dszdepUnjx5onU4FilnJHQAv+6Qpyhs/d7qZunwv77qqbXqERERWof0SrY6G75uW4GvWpdn08mb9Jj6JwlJKVqHpZiB8PBw7t27x++//651KBYp5yR021xQ6324sgfObdI6GqNIW6seHh7OiBEjSEkx30TZK9iHMV39OXj5Pt+v/UvrcBQzUKtWLSpVqkRERIRV1ZWbSs5J6AD+74B7SVgeDk/uah2NUTg5ObF48WJCQ0P5/vvv6dWrFwkJ5rv5c8tKRegVVIypOy+w4cRNrcNRNCaEIDw8nCNHjrBjxw6tw7E4OSuh2zlAhynw+DasfNcql15A31d93LhxjBo1ijlz5tCiRYts3VZtbJ+2KEuFonn4cNFhrt5XZY05Xbdu3cibN68qYcyCnJXQQd8OoMEXcHIlHMjarcaWQAjBp59++qJWvVGjRma73VcuWx0RXauQnCLpPW0vW0/dUh+3czAnJyf69evH0qVLuXLlitbhWJScl9ABgsL1NxytHQF3zmgdjVH17t2bxYsXc/DgQerVq8fNm+a5rOHj4czY7lV4kpBM7+n76DB+NzvO3FaJPYcaMmQIKSkpBt9E2tpZfy+XV3l4DcbXhDxFoP9Gq+jzkpH169fTtm1bvLy82LBhA97e3lqHlK6EpBQW7b/C2M1nufYgnqo+eRnWsDTBJdzfuMeHYtnatGlDVFQUly9ffuMmVdYsZ25w8Tp5ikD7SXDzmH6mbuUaN27MunXruHHjBsHBwRw9elTrkNJlb2tD9+rF2PJRPb5uW4Erd5/SfcqfdJ64h6hzltvXXnlzYWFh3L59m4ULF2odisXIuTP0VBtH6vu8tJ8ClTpqHY3RHTlyhGbNmhEXF8eyZcsICQnROqQMxScms2DfFcZtPcvNh8+oUTwf79YvRY3i7tjYqBm7NZNSUq5cOVxcXNi3b5/W4ZgNNUPPSMjn4FUDIofBnbNaR2N0lSpVIioqCk9PT5o2bcr8+fO1DilDDnY6egX7sO2jEL5sVY5ztx/TbcqfBP9nM1+tPM6xqw+0DlExEiEEYWFhREdH8+eff2odjkVQCV1nC29PAxtbWPW+1ZYypuXt7c3OnTupXr06Xbt2ZfTo0VqH9FoOdjr61PRlx79C+LWLHxU9Xfl9z2VaR+xk81/meaFXyb6ePXuSO3duVcKYSSqhA7gWhfqfw4Xt+nLGHCBv3rysX7+et99+m+HDh/P++++b9V2lqRzsdLTxK8rknoHs+6wh5YrkIXzuQU5ef6h1aIoR5M6dm969e7Nw4UKzrdAyJ5lK6EKIpkKIU0KIs0KIdK8gCiHqCSEOCSGOCyG2GTZMEwjoAwXKw/rPIDFn3Nzi4ODAggULeO+99/jll1/o0qWLRe364upkx5SeVXFxsKXfjH3cemg5sSuZFxYWRmJiIpMmTdI6FLP32oQuhNABY4FmQDmgqxCi3EvvcQPGAa2llOUBy7u6qLOFZv+B+5dhd875eGdjY8PPP//MTz/9xKJFi2jSpInZ3oCUnkKuDkztVZV7TxLpNzOaGw9UUrc2pUuXpkmTJkyYMIHExEStwzFrmZmhVwPOSinPSykTgPlAm5fe0w1YIqW8DCClvGXYME3Etw6UawM7RsODGK2jMRkhBMOHD2fu3LlERUVRq1Yti7pDr0JRVyK6+XPm1iMa/byNxftj1A1JViYsLIxr166ZbWtoc5GZhF4USPt/d8zz59IqDeQVQmwVQuwXQvRM70BCiIFCiGghRPTt27ezFrGxNf4GkLD1O60jMbmuXbuydu1aYmJiqFGjBkeOHNE6pExrULYga9+rw1sFczN80WEGzt5PnOqzbjWaNWtG8eLF1cXR18hMQk+v2Pfl6Y8tEAC0AJoAXwghSv/jm6ScJKUMlFIG5s+f/42DNQk3b6jSEw4vyFGz9FT169d/0eWudu3abNliORuC+Hg4s2BQEJ+3KMvmv27Rd/o+tXmGldDpdAwdOpSdO3dy6NAhrcMxW5lJ6DGAV5rHnsDLG0LGAGullI+llHeA7UBlw4SogeB3AZmj1tLTSlur3qRJE+bNm6d1SJmmsxH0r12cXzr7EX3pLn1n7ONJgkrq1qBPnz44OTmpWXoGMpPQ9wGlhBC+Qgh7oAuw4qX3LAdqCyFshRBOQHXA/HYsziw3L6jUBfbPhDgzXRoystRa9aCgILp168Z///tfi1qXblW5CD939mPfRX1Sf5qQrHVISjblzZuXHj16MHfuXGJjVRuI9Lw2oUspk4AwYB36JL1QSnlcCBEqhAh9/p6TwFrgCLAXmCKlPGa8sE2g1jBIioc/x2sdiWby5s3LunXr6NixIx9++KHF1KqnauNXlJ87+7H3wl36zVRJ3RqEhYURHx/P1KlTtQ7FLKleLhlZ2AvObYb3j4GDq9bRaCYlJYXhw4fzyy+/0LFjR2bNmmVR3e+WHozhg4WHqVnCgym9AnGw02kdkpIN9erV4+LFi5w7dw6dLuf9W6peLllV+wN49hCixmkdiaYsvVa9nb8nP71dmV3n7tB3xj7WHb/BhTuPSU6xnCUk5X/Cw8O5dOkSkZGRWodidtQM/XX+6KtvBzBoOxQoq3U0mps/fz49e/akVKlSrFmzxmz7qqdnUfQVPl16lMRk/c+8s72OYQ1L07eWLzrVudFiJCUl4evry1tvvcXGjRu1Dsfk1Aw9O5r9ALlyw/KhkKLWYLt06cK6deuIiYkhKCjIomrVOwZ6cejfjVk2tCY/vF2J6sXdGbX6JJ0mRnHhzmOtw1MyydbWlsGDB7Np0yZOnrTc2gtjUAn9dZw99En96n7Yk7OXXlKFhISwc+dOhBAWV6vunMsWPy83OgV6MbVXID93rsyZm49o9ut2vl/7F/ceJ2gdopIJAwYMwN7enoiICK1DMSsqoWdGhQ7wVgvY/A3EntM6GrNQsWJFoqKi8PLysoi+6ukRQtDO35MNH9SlcblCTNh2jlrfb+bHdX9x/4lK7OYsf/78dOnShVmzZvHwoeq0mUol9MwQAlqOBttcsGp4juiZnhleXl7s2LGDGjVq0LVrV4urVU9VMI8DY7r6s25YHeqVKcC4reeo9f0WRq8/xYMnqhmUuQoPDycuLo6ZM2dqHYrZUBdF38Se8fr9R7stgtKNtY7GbMTHx9OzZ08WLVrEe++9x+jRo7Gxsdy5wqkbj/h102lWH71Bbgdb+tb0pW8tX1wd7bQOTXlJUFAQd+/e5eTJkxb9M/cm1EVRQwnsB/lK6HumJ6uZWyoHBwfmz5/PsGHD+PXXXy2ur/rL3iqUm3HdA1jzXm2CS7jz66Yz1P5+M2M2nSE+UV0YNydhYWGcPn2aDRs2aB2KWVAJ/U3Y2kPjr+HOadg/Q+tozIql16qnp2zhPEx8J5BV79aienF3Rm84zfBFhy1yWcladezYkYIFC6r+Ls+phP6m3moOPrX17XWf3tc6GrMzfPhw5s2b96Kv+uXLl7UOKdvKF3Flcs9A/tX0LVYduc7cvZZ/TtbC3t6eQYMGsXr1as6dUwULKqG/KSH0PdOf3NVXvSj/YMm16hkJrVOCOqXz89XKE2oPUzMyaNAgdDod48apsmKV0LOiiB9UD4V9k+HgHK2jMUshISHs2LHjRa365s2btQ4p22xsBKM7VcbN0Y6hcw/wMF5dRzEHRYoUoUOHDkydOpXHj3P2DWIqoWdV42+geD1YOQwu79E6GrOU2lc9tVbdkvqqv4qHSy5+7eLPxTuPqT5qE2FzD7D22HWSki2nC6U1Cg8P58GDB8yZk7MnWKpsMTue3oPJDSD+AQzcot/tSPmHe/fu0a5dO7Zt28aPP/7I8OHDEcKye6ccuHyPP/bHsPbYDe4+TuDtAE9+6mi5e7pYOiklAQEBJCYmcuTIEYv/+cqIKls0Fse80G0BpCTCsiFaR2O28ubNy9q1a+nYsSMfffSRxfVVT08V77x8264iez9twMA6xfljfwyb/7qpdVg5lhCCsLAwjh07xrZt27QORzMqoWeXRymo9wlc3AGXdmsdjdl6uVa9a9euJCRY/u31tjobhjcuTemCLny65JhaV9dQ165dcXd3z9EljCqhG0KVXuBcALb9oHUkZi21Vv2HH35g4cKFtG/fnqdPn2odVrblstXx49uVufUonlGRqvufVhwdHenfvz/Lli2zinLZrFAJ3RDsnaDmu3B+C1zZq3U0Zu+jjz5iwoQJrF69mhYtWhAXF6d1SNlW2cuNgXVKsCD6Cgujr5CiNs/QxODBgwEYPz5nbh2pErqhBPYFJ3c1S8+kQYMGMWvWLLZv307Dhg25c+eO1iFl27CGpajk6cq//jhCi992su74DXVXqYkVK1aM1q1bM3nyZJ49e6Z1OCanErqh2DtDcDic3aDvna68Vo8ePfjjjz84fPgwwcHBnD9/XuuQssXBTseSwcGM7lSZpwlJDJq9n9A5+0lIsuwLwJZm8ODBxMbGsnTpUq1DMTmV0A2pan995cum/1MtdjOpbdu2bNy4kTt37hAUFMT+/Zb9y9BWZ0P7Kp5s/KAuI5qVYd3xm4TPO0CiqlM3mYYNG+Lj48PkyZO1DsXkVEI3pFy5IeQzOL8VDs3VOhqLUbNmTXbt2oWjoyN169Zl7dq1WoeUbbY6G0LrluDLVuVYd/wm7847qJK6idjY2NCvXz82b97M2bNntQ7HpFRCN7TAfuAdDOs+gYfXtY7GYpQtW5aoqChKlSpFq1atrGbTgj41ffmiZTnWHLvB4DkHePwsSeuQcoQ+ffqg0+mYMmWK1qGYlErohmZjA20iIOkZrPpALb28gcKFC7Nt2zbq1atH7969GTVqlFVcVOxXy5f/a1OezX/d5O0JUVy7b/mlmuauaNGitGjRgunTp1vF/Q6ZpRK6MbiXgPqfw6nVcGyx1tFYlDx58rBq1Sp69OjB559/zpAhQ0hKsvxZbc8gH6b2rsqVu09oM3YXR2Luax2S1Rs4cCC3bt1i5cqVWodiMiqhG0uNIVA0UN+868ZRraOxKPb29sycOZMRI0YwYcIE2rdvz5MnT7QOK9tC3irAkiHB2Ots6DsjmluPLHdXJ0vQtGlTPD09c9TFUZXQjcVGB51ng0Me+L0TPLiqdUQWxcbGhu+++46IiAgiIyOpX78+t2/f1jqsbCtdMDfT+1Ql7lki7y84RLK6AclodDod/fr1Y/369Tnm4qhK6MaUpwh0XwQJcfB7R31XRuWNDB06lMWLF3P48GFq1qxp8bXqoE/qX7Uuz66zsYzfmjMSjVYGDhyITqfLMXeOqoRubAXLQ6dZcOcUzO8OieqC2Jtq166dVdWqA3QK9KKNXxFGbzjNn+djtQ7HaqVufjFt2rQcsfmFSuimUCIE2o6HizthfjdIVGunb8raatWFEIxqVxHvfE58suSoWnoxovDwcO7fv58jNr9QCd1UKnWCNmPh3BZY0F1f1qi8kZdr1WfMmKF1SNniksuWj5uW4fydx0QeuaZ1OFYrODgYf39/IiIirKIMNiOZSuhCiKZCiFNCiLNCiBEZvK+qECJZCPG24UK0Iv7dodWvcHYjLOwFyZZfjmdqaWvV+/TpY/G16k3KF+KtgrkZs+mMmqUbSU7a/OK1CV0IoQPGAs2AckBXIUS5V7zve2CdoYO0KgG9oPlPcHoNrP5Q3XiUBS/Xqg8ePNhia9VtbATvNijFuduPWXVU3VlsLDll84vMzNCrAWellOellAnAfKBNOu8LBxYDtwwYn3WqNgBqDoP902HXL1pHY5Hs7e2ZNWsWI0aMYOLEiXTo0MFia9WbVShEqQIu/LbpjOqjbiQ5ZfOLzCT0osCVNI9jnj/3ghCiKNAOmJDRgYQQA4UQ0UKIaGuoKc6WBl9ChQ6wcSQc/UPraCySEOJFrfrKlStp0KCBRfZVt7ERhDcoxZlbcaw+pmbpxpK6+cWECRmmKYuWmYSe3vbZL08jfgE+llImZ3QgKeUkKWWglDIwf/78mQzRStnY6CtfitWEpaFwdpPWEVms1Fr1Q4cOWWxf9RYVC1MivzOjVp3kyl3L/KRh7tJufhEfb52VZplJ6DGAV5rHnsDLl+QDgflCiIvA28A4IURbQwRo1WxzQZe5kL8MLOgBV/ZpHZHFSlurHhwczIEDB7QO6Y3obARjuvrzJCGZLpP2EHNPJXVjCA8P586dOyxYsEDrUIwiMwl9H1BKCOErhLAHugAr0r5BSukrpfSRUvoAfwBDpJTLDB2sVXJ0gx6LwaUg/P423DyhdUQWK7VW3cHBgTp16lhcrXr5Iq7M6VedR/GJdJ28h6uqK6PBhYSEUK5cOX777TeLro56ldcmdCllEhCGvnrlJLBQSnlcCBEqhAg1doA5Qu6C0HMZ2DnC7LZwS+0cn1WWXqte0dOVOf2rc/9JIu3G7mLLX6rGwJBSSxj379/Pn3/+qXU4Bie0+i0VGBgoo6OjNRnbbN36C2a1geQEeGcpFPHTOiKL9fDhQzp06MDGjRv5+uuv+eyzzxAivctB5unEtYcMW3CQ0zfj6BjgyRetypHHwU7rsKxCXFwcRYsWpWXLlvz+++9ah/PGhBD7pZSB6b2m7hQ1JwXKQJ/V+g2nZ7aCy9Y3gzCVtLXqX3zxhcXVqpcrkoeV4bUYUq8Eiw/E0HbsLuLUbkcG4eLiQp8+fVi0aBE3btzQOhyDUgnd3LiXgL5rwTm/frZ+POftXG4oqbXqn3zyCRMnTrS4vuq5bHX8q2kZZvatxsU7j/l0yVGrXPfVQlhYGImJiUyaNEnrUAxKJXRz5OoJfddB4UqwqDds+Q5S1AbDWSGE4Ntvv33RV90Sa9Vrl8rP+w1Ls+LwNRZGX3n9NyivVbJkSZo1a8aECROsaos6ldDNlUt+6LUS/LrDtv/AH71Vl8ZssPRa9SEhJalZ0p0vVxzn1I1HWodjFcLCwrh+/TpLlizROhSDUQndnNnm0ndobPQ1nFgO8zpDgvX3dDYWS+6rrrMR/NzZD5dcdgyde4AnCWo9PbuaNm1KyZIliYiI0DoUg1EJ3dwJATXf1d9VemE7zG6vdj7KBkvuq14gtwO/dvHj3O04vlx+XOtwLJ6NjQ1Dhw5l165dHDx4UOtwDEIldEvh1w3engZXo2Fma3imPnZnlSXXqtcs6UFYSEkW7Y9hyYEYrcOxeL1798bJyclqujCqhG5JyreDzr/DjaOweACkZNg6R8lAal/1kJAQi+ur/l6DUlTzycfny45x7nac1uFYNDc3N3r27MncuXMt7mJ5elRCtzRvNYWm/9H3U9/8tdbRWLQ8efIQGRlpcX3VbXU2jOnqTy5bG4bMOcD9J9ZTpaGFsLAwnj17xtSpU7UOJdtUQrdE1QZAYF/Y+TMcts4mQ6bycl91S6lVL+TqwG9dq3DhzmPembqXB08TtQ7JYpUvX56QkBDGjRtnEb/QM6ISuiUSApr9AD61YflQfU91taaeZWn7qkdGRlK/fn2L+Phdq5QHE96pwl83HtJz6p8qqWdDeHg4ly9fZuXKlVqHki0qoVsqnR10ng0V39bP1MdUgQOz1bp6NgwdOpQlS5Zw+PBh6taty/Xr5r/ZRP0yBRnfPYAT1/VJ/U6c2nw8K1q1aoW3t7fFlzCqhG7JHPNCuwnQfzPk9YEVYTCpHlzcpXVkFqtt27asXr2aS5cuUbt2bS5duqR1SK/VsJw+qZ+6+Yg2EbvUjUdZYGtry5AhQ9i8eTPHj1tuSahK6NbAMwD6rYcOU+HJXZjRHBb2hMexWkdmkUJCQtiwYQN37tyhdu3anDp1SuuQXqthuYIsHBREYnIKHcbvVm13s6Bfv37kypXLomfpKqFbCyH0yy9h+yDkczi1FiaHqA0zsigoKIitW7cSHx9PzZo12bNnj9YhvVYlTzeWh9WkmLsTA2ZFc+yqugHtTXh4eNCtWzdmzZrF/fv3tQ4nS1RCtzb2TlD3I+izBpKewdRG8NdqraOySH5+fuzevRs3Nzfq16/PihUrXv9NGivs6sjc/jXI62zPh4sOk5Ckmrq9ifDwcJ48eWJRN5ulpRK6tfIMgIFbwKMUzO8Gqz9SlTBZULJkSXbv3k2FChVo164dEydO1Dqk13J1smNU2wr8deMR47ae1Toci+Lv70/NmjUZO3YsKRbY4VQldGuWp4h+pl5tIOydDGOrq9l6FhQoUIAtW7bQtGlTQkND+eKLL8z+rtLG5QvRunIRIjaf5eT1h1qHY1HCwsI4e/Ys69at0zqUN6YSurWzc4TmP0D/jeDgBvO7wpKBqsHXG3J2dmb58uX079+fb775hj59+pCYaN513yNbl8fNyY6P/jhMUrLlzTa10qFDBwoXLmyR/V1UQs8pPANh0Dao9wkc/QPG14SLO7WOyqLY2toyadIkRo4cycyZM2nVqhWPHpnvMlY+Z3tGti7PsasPWXLgqtbhWAw7OztCQ0NZs2YNZ86c0TqcN6ISek6is4N6I/Qljjo7mNESdv0KZr58YE6EEHz55ZdMnTqVjRs3Uq9ePbPel7JFxcL4e7sxesNp4hPVTWeZNXDgQOzs7Bg7dqzWobwRldBzIs9ACN0J5dvChn/Dmo/VHaZvqG/fvqxYsYK//vqL4OBgTp8+rXVI6RJCMKJpGW48jGf6rotah2MxChUqRMeOHZk+fTpxcZbT0VIl9JzK3hk6TIMaQ2HvRP3epYlPtY7KojRv3pytW7cSFxdHcHCw2daqVy/uToMyBRi39azqzPgGwsLCePjwIbNnz9Y6lExTCT0ns7GBpt9C41FwcoW+bcD1w1pHZVGqVq1qEbXq/2pahrhnSYzbek7rUCxGjRo1CAgIICIiwuyrmlKphK5AcBj0WAJP78PkBrDjv2oJ5g28XKs+YcIErUP6h7cK5aZDFU9m7L7Igcv3tA7HIgghCA8P58SJE2zZskXrcDJFJXRFr2QDGBIFZZrDpv+DiXX0e5gqmZJaq96sWTMGDx5slrXqHzctQxFXB3pN3ctBldQzpXPnznh4eFhMCaNK6Mr/OOWDjjOh4wyIfwgzW8H87nDtkKqEyQRnZ2eWLVv2ola9b9++ZlWrnj93LuYNrEE+F3t6Tt3L4Sv3tQ7J7Dk4ODBgwABWrFhhEZ03VUJX/k4I/d6lYXuh/hdwbgtMqgsTasGe8XD/itYRmrW0teozZsygVatWZlUlUdjVkXkD9L1eekz9k2v31YXw1wkNDQVg3LhxGkfyekKrj4WBgYEyOjpak7GVN/D0HhxbDAfnwLWD+ufy+uh3SyrZUP+Vy0XTEM3V1KlTGTRoEJUrV2bVqlUUKlRI65BeuBT7mIajt9G5qhfftK2odThmr0OHDmzdupWYmBgcHR01jUUIsV9KGZjea2qGrmTMMS9U7Q8Dt8LQvdD0eyhQXl8Vs6gX/FgC5nXV94hRyzJ/069fP5YvX/6iVt2c+qoXc3emY6AXC/fFcP2BmqW/Tnh4OHfv3mXevHlah5KhTM3QhRBNgV8BHTBFSvmfl17vDnz8/GEcMFhKmWH9m5qhW7jkJLgcBSdX6r8eXYPCfhDyGZRqpF+6UQDYt28fLVq0ICUlhZUrVxIUFKR1SADE3HtCyE9b6VbNm6/aVNA6HLMmpaRy5crodDoOHDiA0PDnO1szdCGEDhgLNAPKAV2FEOVeetsFoK6UshLwNTApeyErZk9nC7619Y2/hh2FNmPh6V2Y21HfJyZqHDw2/42WTaFq1apERUW9qFVfvny51iEB4JnXibcDPJm37wo3H8ZrHY5ZE0IQFhbGoUOH2L17t9bhvFJmllyqAWellOellAnAfKBN2jdIKXdLKVProPYAnoYNUzFrOlvw7wHhB6D1b2CbC9Z9Av8tAwt6wOl1+hl9DlaiRAl2795NxYoVad++vdnUqg+pV5KUFMl4dcPRa3Xv3h03NzezLmHMTEIvCqQtbYh5/tyr9APWpPeCEGKgECJaCBF9+/btzEepWAadHVTpqd9YY3AUVB8El6Jgbif4ubx+k42TK/X7nuZAL9eqf/7555rXqnvlc6J9laLM23uZW2qWniFnZ2f69u3L4sWLuXbtmtbhpCszCT29xaJ0fwqFECHoE/rH6b0upZwkpQyUUgbmz58/81EqlqdgOWgyCob/BV3mQtEqcGC2fsb+Q3GY2xnict4v9bS16qNGjTKLvupDQ0qSmJzC9N0XNY3DEgwZMoTk5GSz3bkqMwk9BvBK89gT+MevJyFEJWAK0EZKqbabV/R0dlCmBXSdByMuQ5+1UOdDfX37xNr6GXwOY2591Yu5O9O0QiHm7LlE3LOcvTT2OiVKlKB58+ZMnDiRhATza3SWmYS+DyglhPAVQtgDXYC/dSASQngDS4B3pJTm2UdU0Z6tPRQLgvqf63dQsnWAGS1g8zfw0Dw/whpLal/1KVOmmEVf9YF1SvAoPon5ey9rFoOlCA8P5+bNmyxatEjrUP4hs2WLzYFf0JctTpNSjhJChAJIKScIIaYAHYDUe2OTXlVWk0qVLSrEP4CVw+D4EhA2UKIB1AjV36yUg6xevZqOHTtSsGBB1q5dS+nSpTWJo9PEKGLuPmHbv0Kw06lbVF4lJSWFsmXLkjdvXk1aJmf7xiIp5WopZWkpZQkp5ajnz02QUk54/vf+Usq8Ukq/518ZJnNFAcDBFTpO11fH1PoAbp2AOR1g8yiwwB3Xs8pc+qoPqlOcaw/iWXXkuibjWwobGxuGDh3Kn3/+yb59+7QO52/Ur2FFe+4loMEX8O5Bffnj9h9gUU94Zj49UIwttVY9b968mvVVD3mrAKUKuDBh2znNq2/MXe/evXFxcSEiIkLrUP5GJXTFfNjmgtYR0OQ7+GuVvoXv/pmQmDPK6dLWqmvRV93GRjCgTnH+uvGIaWq7ugzlyZOHXr16MX/+fG7duqV1OC+ohK6YFyEgaAj0WAx2TrDyXX0N+9pPIXo6nN1k1RdQ8+fPz+bNmzWrVW/vX5TG5QrydeQJZkVdNNm4lmjo0KEkJCQwZcoUrUN5QXVbVMyXlHBxB0SN1SfylNR6bQFvNYegoVAs2Cr7xiQlJTFkyBAmT55Mr169mDx5MnZ2diYZOyEphSG/H2DjyZt807YCPWoUM8m4lqhRo0acPHmSixcvYmtra5IxM7ooapoIFCUrhADfOvqvlGR4dB3uX4Zzm2HfVDi1CvKXBa+qUKgSeNeAQtbRCtbW1paJEyfi6enJl19+yY0bN1iyZAlOTk5GH9ve1oZx3asweM5+Pl92jKJujoSUKWD0cS1ReHg4bdq0YdmyZbz99ttah6Nm6IqFSngCR+bD8WVw44i+bztA3Y+h7gj9BthWYsqUKQwaNIiaNWsSGRlJnjx5TDLus6RkWozZSUJSCuvfr4ODnc4k41qS5ORkSpYsSbFixdi6datJxlT90BXrY+8EgX2h1wr41wV4/zj4dYdt3+v7tCc81jpCg+nfvz9z584lKiqKhg0bcveuaXrh5LLVMbJVeS7ffcLk7edNMqal0el0DBkyhG3btnHkyBGtw1EJXbECQoCrp76Fb5Nv4a9ImNoEru7XOjKD6dy5M0uWLOHIkSPUrVuXmJgYk4xbq5QHzSsWYuzWs8Tce2KSMS1Nv379cHBwYOzYsVqHohK6YkWE0F8o7bYI4m7C5PqwNNRqqmJatWrFqlWruHTpEkFBQRw/ftwk437WQr/9wTeRJ00ynqXJly8f3bt3Z86cOdy7d+/132BEKqEr1qdUQwjfD7Xe1++HOsYf5nWDQ/MsvnVvgwYN2L59O0lJSdSqVYvt27cbfcyibo6EhZRk7fEbLD901ejjWaLw8HCePHnCtGnTNI1DXRRVrNu9i/rdk1K3yRM68KkFZVtBmZaQp7DWEWbJxYsXadq0KRcuXGDOnDl07NjRqOM9S0qmx5Q/OXD5PhFd/WlW0TL/uxlTnTp1iImJ4cyZM+h0xruArC6KKjlXXh/9NnnvH4f+m6Hmu/olmNUfwugyMLkB7PwZYi1rxx4fHx927dpF1apV6dy5M7/88otRx8tlq2N6n2r4ebkRPu8ga49p1xnSXIWFhXHhwgXWrEl3fx+TUDN0JWe6fQpOroCTkXD9kP65cm2h0Vf6XwIW4unTp3Tv3p2lS5cyfPhwfvjhB2yMWLL5KD6RXtP2ciTmAdP7VKV2KbVRTarExER8fX2pUKECa9euNdo4aoauKC/L/xbU+QgGbYNhx/S162fWQ0Q12DgSLu6C+1f0NzSZMUdHRxYtWkRYWBj//e9/6d69O8+ePTPaeLkd7JjRtxol8rswbP4htbl0GnZ2doSGhrJu3TpOnTqlSQxqhq4oqR5chU1fwZEF/3tOZ6/vz+7fA0o11u/AZIaklPz44498/PHHhISEsHTpUlxdXY023tlbj2j12y4qe7nye/8a6Gysr/1CVty8eRNvb28GDRrEmDFjjDJGRjN0ldAV5WX3r0DsGbh3Sb80c3yJvgzSyUO/N6qbN7gV0/eRKVLFrO5KnTNnDn369KFs2bKsWbOGokUz2s89e/7YH8OHiw7zXoNSvN9Im005zNE777zD8uXLuXr1Krlz5zb48VVCV5TsSE6Csxvh6CK4c1rfTyb+vv613EX0e6b6d4ci/pqGmWrjxo20b98eNzc31qxZQ/ny5Y021gcLD7H04FW+aFGO7jW8yWWr2gPs3buX6tWrExERwdChQw1+fJXQFcXQHsfqk/zJFfpOkElPwTtYf2NT6SaaL80cOnSI5s2b8/TpU5YtW0bdunWNMs7jZ0n0nxlN1PlYCrs6MCSkJJ0DvbC3NZ9PLVqoVq0ajx494sSJEwgDdwNVCV1RjCn+IRycDXsmwIPLoMsFBcpC4UpQtrV+DV6DFr+XLl2iadOmnD9/3qi16lJKdp69w88bTnPg8n2q+eRjwjsB5HO2N8p4lmD27Nn07NmT9evX06hRI4MeWyV0RTGF5CQ4sw4uR8H1I3D9sH5ppmgghHwKJeqbPLHfvXuX1q1bs3v3bn7++Wfee+89o40lpWT5oWt8vPgIBfLkYlqvqpQqaPg1ZEvw7NkzvL29qVGjBsuXLzfosVXZoqKYgs5Wv57e+Bt9F8gPz0DLX+DRDZjTHqY0gGNL9InfRPLly8eGDRto164dw4YN48MPPyTFSBtwCyFo61+U+QNr8DQhhfbjdrP77B2jjGXucuXKxcCBA1m5ciUXLlww2bhqhq4oxpb0DA7OgagIuHseXL30bQfyFtNXyxTxN3oLguTkZIYNG0ZERARdunRhxowZ5MqVy2jjXb3/lL7T93Hp7mOm9apKcEkPo41lrmJiYvDx8eH999/nxx9/NNhx1ZKLopiDlGQ4vQ72jNO39k183o5W2OiXY/y667fWs3MwyvBSSn744QdGjBhhklr12LhndJv8J5fuPmZ672oElXA32ljmqlOnTmzcuJGYmBiD7TalErqimBsp4UmsvnnY6bX6TpAPY8DWEbyrg09tfZ17wQrgYNgdiubMmUPfvn0pU6YMq1evxtPT06DHT+tO3DO6TtpDzL2njO9RhXpv5ayt7LZv307dunWZPHky/fv3N8gxVUJXFHOXkgznt+rbD1zYAbfS9DrPVxy8g6ByFyhWyyA3MqXWqru6urJ27Vqj1qrffvSMHlP+5NTNR3Sv7s2nzcvinCtnbGcspcTPzw/Ql5IaooRRJXRFsTSP7+iXZa4fgRuH4fw2ePZQv+Zevp1+3b1wJcjrm+XKmUOHDtGsWTPi4+ONWqsOEJ+YzH/Xn2LKzgsUdXPkoyZv0bBswRyR2KdMmcKAAQPYtm0bderUyfbxVEJXFEuX8ES/td7BOXBxJ8jnTcNy5YFCFaFQJf2fhStB/jKZvrHp4sWLNGvWjPPnzzN79mw6depkxJOA6It3+dcfRzh/5zG5bG2oX6YAoXVLUNnLzajjaunJkyd4enrSsGFDFi5cmO3jqYSuKNYk8SncOgk3jjyfwR+Fm8f+d5FVZw8eb/2viiafr34tvlAFyPXPuvC0teqjR49m2LBhRg0/JUUSfekeq45cI/LIdRKSU1g6JJiSBay3Zv2jjz7i559/5uLFi9m+ZqESuqJYu5Rk/SYdN57f0HT7L31zsfuX9W0JABD6xmJO7uDgCs75oWQDKN2Epzi86Kv+wQcf8OOPPxq1r3qqq/ef0iZiF472NiwbUhN3F+OVUmrpwoULlChRgk8//ZRvvvkmW8dSCV1Rciop9Ts03TiqT/Z3TsPTexD/QJ/s426CjS341Ca5QmeGTdlOxPgJJqlVT3Xoyn06T4yiQlFXfu9fHQc762zw1bp1a/bs2cPly5dxcMh6aapK6Iqi/FNKClw7qG8wdmIZ3LuItM/Nj6eL8fHM3dQLrsrSFatwczf+rkSrj15nyO8HKObuRIWirpQukJuAYnmpUTwftjrruKF9w4YNNG7cmFmzZvHOO+9k+TjZTuhCiKbAr4AOmCKl/M9Lr4vnrzcHngC9pZQHMjqmSuiKYkZSUuDybjj4O5xYzu/779Nn+VPK5Ncxr0dRyvsUAAc3cC2qX5d389L3h3dwBUc3/Z8ObvqLtLqsVa4sP3SVlYevcfpmHFfuPUFKyOdsT9MKhehQxZOAYnkNecYmJ6WkbNmy5MmTh71792b5ONlK6EIIHXAaaATEAPuArlLKE2ne0xwIR5/QqwO/SimrZ3RcldAVxUwlJ0HsWTYtn0vH4T/y8PEzQhuVZmTLYnik3IYHVyA54dXfb+/yvwSfu9Dzi7Pe4Fzgf8lfl2Ypx8FV/3qaO2QfP0tix5k7rDp6nU0nb/IkIZnapTx4v1FpqnhbbmKPiIggPDycPXv2UL16hinylbKb0IOAkVLKJs8ffwIgpfwuzXsmAlullPOePz4F1JNSXn/VcVVCVxTzFxsby5dffsmECRNwcnLCy8tL/0JKkv5CrEzWz+5lMsiU5889f5ySrH9fcuL/yiwzYmMLIr31c0myhOQUCVIavL+4KaVIyYXYBMqX9OTI6StZOkZGCT0zn42KAmlHjkE/C3/de4oCf0voQoiBwEAAb2/vTAytKIqW3N3diYiIYPDgwYwePZqHDx9m7UApSfomZcmJkJKoT/qpkhL0JZcJT/SvvYIEniQkk5BsnG6RpuKeN5HmzRsY5diZSejp/Tp8eVqfmfcgpZwETAL9DD0TYyuKYgbKly/P1KlTtQ5DeY3MXD6OAbzSPPYErmXhPYqiKIoRZSah7wNKCSF8hRD2QBdgxUvvWQH0FHo1gAcZrZ8riqIohvfaJRcpZZIQIgxYh75scZqU8rgQIvT56xOA1egrXM6iL1vsY7yQFUVRlPRkqmBUSrkafdJO+9yENH+XwFDDhqYoiqK8Ceu4BUtRFEVRCV1RFMVaqISuKIpiJVRCVxRFsRKadVsUQtwGLmXx2z2AOwYMxxKoc84Z1DnnDNk552JSynRbYGqW0LNDCBH9ql4G1kqdc86gzjlnMNY5qyUXRVEUK6ESuqIoipWw1IQ+SesANKDOOWdQ55wzGOWcLXINXVEURfknS52hK4qiKC9RCV1RFMVKmHVCF0I0FUKcEkKcFUKMSOd1IYQY8/z1I0KIKlrEaUiZOOfuz8/1iBBitxCishZxGtLrzjnN+6oKIZKFEG+bMj5jyMw5CyHqCSEOCSGOCyG2mTpGQ8vEz7arEGKlEOLw83O26K6tQohpQohbQohjr3jd8PlLSmmWX+hb9Z4DigP2wGGg3EvvaQ6sQb9jUg3gT63jNsE5BwN5n/+9WU445zTv24y+6+fbWsdtgn9nN+AE4P38cQGt4zbBOX8KfP/87/mBu4C91rFn45zrAFWAY6943eD5y5xn6NWAs1LK81LKBGA+0Oal97QBZkm9PYCbEKKwqQM1oNees5Ryt5Ty3vOHe9DvDmXJMvPvDBAOLAZumTI4I8nMOXcDlkgpLwNIKS39vDNzzhLILfS7QLugT+hJpg3TcKSU29Gfw6sYPH+Zc0J/1cbTb/oeS/Km59MP/W94S/bacxZCFAXaAROwDpn5dy4N5BVCbBVC7BdC9DRZdMaRmXOOAMqi377yKPCelNKyd4TOmMHzV6Y2uNCIwTantiCZPh8hRAj6hF7LqBEZX2bO+RfgYyllsn7yZvEyc862QADQAHAEooQQe6SUp40dnJFk5pybAIeA+kAJYIMQYoeU8qGRY9OKwfOXOSf0nLg5dabORwhRCZgCNJNSxpooNmPJzDkHAvOfJ3MPoLkQIklKucwkERpeZn+270gpHwOPhRDbgcqApSb0zJxzH+A/Ur/AfFYIcQEoA+w1TYgmZ/D8Zc5LLjlxc+rXnrMQwhtYArxjwbO1tF57zlJKXymlj5TSB/gDGGLByRwy97O9HKgthLAVQjgB1YGTJo7TkDJzzpfRfyJBCFEQeAs4b9IoTcvg+ctsZ+gyB25Onclz/jfgDox7PmNNkhbcqS6T52xVMnPOUsqTQoi1wBEgBZgipUy3/M0SZPLf+WtghhDiKPrliI+llBbbVlcIMQ+oB3gIIWKALwE7MF7+Urf+K4qiWAlzXnJRFEVR3oBK6IqiKFZCJXRFURQroRK6oiiKlVAJXVEUxUqohK4oimIlVEJXFEWxEv8Ph1Ba0kw2pvQAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(df_scores[\"threshold\"], df_scores[\"tpr\"], label=\"TPR\")\n",
"plt.plot(df_scores[\"threshold\"], df_scores[\"fpr\"], label=\"FPR\")\n",
"\n",
"#plt.plot(df_rand[\"threshold\"], df_rand[\"tpr\"], label=\"TPR\")\n",
"#plt.plot(df_rand[\"threshold\"], df_rand[\"fpr\"], label=\"FPR\")\n",
"\n",
"plt.plot(df_ideal[\"threshold\"], df_ideal[\"tpr\"], label=\"TPR\", color=\"black\")\n",
"plt.plot(df_ideal[\"threshold\"], df_ideal[\"fpr\"], label=\"FPR\", color=\"black\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "faf97765-3835-466c-a105-865fcff92777",
"metadata": {},
"source": [
"* Comparing two models is not always intuative\n",
"* These two models have different thresholds\n",
" * Compare TPR and FPR"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "c9a179e8-4b85-4df6-8ad9-eebe1c78901b",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAE9CAYAAABtDit8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA20ElEQVR4nO3dd3hUZfr/8feTSSMJgZACoSb0GlpoUhVRsGFvIIIFXVd23d2vq64o+tO1rburqyKLiKCguCJNBQtNQHrooYYeAqmQ3mbm+f1xIkYkBTIzZ8r9uq5cZM6czNwHmE+ec85TlNYaIYQQVfMzuwAhhHB3EpRCCFEDCUohhKiBBKUQQtRAglIIIWogQSmEEDXwN7uASxUVFaXj4uLMLkMI4WWSkpKytNbRF3vO44IyLi6OrVu3ml2GEMLLKKWOV/WcnHoLIUQNJCiFEKIGEpRCCFEDCUohhKiBBKUQQtRAglIIIWogQSmEEDVwWlAqpWYqpTKUUnuqeF4ppf6jlEpRSu1SSvVyVi1CCFEXzmxRzgJGVvP8KKBdxddE4H0n1iKEEJfNaSNztNZrlFJx1ewyGvhYG1Osb1RKNVRKxWqtTzujnvTCdNaeWuuMlxZCOJDWGrsGm11js2usFX/a7HZsWmOzcf57q11jtWlj+8/7Wa1EZ25gxLDJdGveziE1mTmEsRlwstLj1IptvwlKpdREjFYnLVu2vOQ3KiovYvy340ktSL28SoUQHid21//o1vxZh7yWmUGpLrLtogv4aK2nA9MBEhMTL3mRnze2vMGpglO8N/w9OkR0uNQfF8LtFZXZyC4sJTO/lKyCUrLyyygos1JSZqPEaqOk3E5Jua3SV8Vja8XjMjslVhv2OiyhpRQEWSwE+PsRZFEEBvgR6G8h0OJHoL8iwM/PeM7fD38/PwIsCn+L8WeA5Zdt5x9b/AiwXLDNr2KbvyLAT/2yj58f/n52Yje8SOiRZRT2+zNRQ//ksL9fM4MyFWhR6XFzIM3Rb7LqxCq+PPQlD3R9gCHNhzj65YVwKq01GfmlnMgpIj2vhPS8UjLySs5/n55fQkZeKQWl1ov+fL0ACyGBFoIDLNQLDCAkMJjgAAvRIRbqBVR8BVb6M/CX7cEBFoL8/Qj09yPI30JQgBFyQf6Vt/sRVLGfv59CqYu1f5zMZoWU5bB+GhxZBVdOhqFPOvQtzAzKJcDjSql5QD8g1xnXJ9/Z8Q5tG7bl8R6PO/qlhXAYm12TeraIlIyC81+HMgo4nFlAfsmvQzDQ34/G4UE0rh9MpybhDG0fROPwYGLqG382Dg8iun4w4cH+5gSXq2Qfhu1zYMenUHAGQqPh2ldgwO8d/lZOC0ql1GfAMCBKKZUKTAECALTW04ClwHVAClAETHBGHUXlRfSK6UWAJcAZLy/EZTlXVMbWY2fZcjyHrcfOsudULqVW+/nno+sH0TY6jJt7NKNtTBhxUaE0qQjBBvUCvDsAa1J8Dr7+EyQvAOUH7a6BnvdB+2vBSZ9zZ971vqeG5zXg+OgXwk2dLSzjvVUprDmUycH0AgACLIpuzRowtn8r2jcOo21MGG2j69MgRH6xX9TJLTD/AchPgyFPQuIDEN7U6W/rcRP3CuFptNbMT0rl1WX7yS0uZ1DbKEb3aEZiqwi6t2hIcIDF7BLdn90O6/8DK18ygnHCt9Cij8veXoJSCCc6mJ7P5IV72Hwsh8RWEbx8S1c6Ngk3uyzPUnwOFj4KB5dBp5vgpnegXkOXliBBKYQDaK0pLLORkVdCZn4pGfml7Dh5jtnrjxEW7M/rt3Xjjt4t8PPz4WuLlyPzAHx6J+Smwqg3oO9Eox+Si0lQClFLdrsmLbeYw5mFHK64I304s4C0c0Y4FpfbfvMzd/RuzjPXdaJRaKAJFXuB7ydDSR6MXwot+5lWhgSlEBehteZgegHL96Wz/0w+hzMKOJJVQEn5L3emG9QLoE10KD1aNCSmfhDR9YOICQ8iOiyYmIruO3JTpg7Ki+HoGug93tSQBAlKIc7TWrP7VC7L9pzh2z1nOJpVCECLRvVoEx3GgDaRtI0Jo010GG2iQ2kUGujb3XSc7egasJZAuxFmVyJBKXxLqdVGVkEZWfnGcL/MglKy8ktJyy1hzcFMTp0rxuKnuKJNJA8OiueaLo2JqR9sdtm+acenUC8C4gabXYkEpfAuxWU2vtqVxmebT3DgTP6vntOai15HBOM0OrFVBE9c3Y4RnRvTMESuKZqqMBv2fwN9HgL/ILOrkaAU3uFgej6fbjrBl9tSyS+x0iY6lLv7tMRywYyr4cEBRFdcT4wKM/6MDAskyF/6MrqVXZ+DvRx63Wd2JYAEpfBgJeU2lu4+zaebTrD1+FkCLX6M6taEe/u2pG98I7l+6Km0hm0fQ9Ne0LiL2dUAEpTCA+WXlPPBmiN8vPE454rKiYsM4W/XdeT23i2kG443OJUEmfvghn+bXcl5EpTCY5RabczdeIJ3V6WQU1jGtV0aM25AHANaR0pHbm+y7WMICIGut5tdyXkSlMLt2e2axTtP8c/vD5J6tpgr2kTy9KiOJDRvaHZpwtHKCmHPAuh8MwS7z1BPCUrh1tYeyuSVpfvZdzqPLk3DeeWWbgxuFyXXH71V8iIoy3ebmzg/k6AUbulIZgGvLN3H8n0ZtGhUj7fv7sGNCU3lFNvbnN5pXJPMSoHsQ5C6BSLbQssBZlf2KxKUwq3kFpfzzopDzN5wjCB/C0+P6siEgXHSfccbbZkB3/zF+N6/nhGQrYdB/8dMmfiiOhKUwi1orflq12leXJJMTlEZd/ZuwV+ubS+jYrzVhqnw3TPQfhRc9w8IbwZ+fjX/nEkkKIXpTucWM3nhHlbsz6B78wbMfqAvXZs1MLss4Syndxkh2elGuG0m+Lt/ly4JSuFydrvmdF4Jx7MK2Zmay3urUrDa7Uy+vhMTBsZjkeuQ3m3V3yG4Adz0rkeEJEhQChcotdrYeCSH5XvT2Xw0h2PZhb9aSGtg20hevSWBlpEhJlYpXOLERjj4LVz1nMtnKa8LCUrhNEnHc5ix9ihrDmZSWGajXoCFfq0bMaR9FHFRocRHhtIqKpSmDYKlu48v0Bp+eB7CmkD/35ldzSWRoBROseFwNhNmbSYsKIDRPZsxolNjBrSJlIW0fNn+b+DkJrjxbQgMNbuaSyJBKRwu6XgOD87eQouIEOZN7E9kmPnTZAmTFeXAt89AVHvoMdbsai6ZBKVwqJ0nzzF+5haahAcz9+F+EpLCWGp2wcNQcAYe+BYsnhc7nlexcFvJabmMm7mZhqEBzH24n/SBFIY1b0DKcmM2oGa9za7msrhvD0/hUfacyuXeDzYRGmjh04f6E9ugntklCXdwaDmsfg263wu9J5hdzWWTFqWos92puYz9cBNhQf7Mm9ifFo2km48Azp2ABQ8Zk+9e/0+3G5Z4KaRFKeok6fhZxszYKCEpfs1ug/kPGn/e+TEEevb/C2lRistSbrPz7soU3l2VQtOGwXz2cH+aR3j2h0E40MapkLoZbv0AItuYXU2dSVCKS5aSUcCf/7eDXam53NqzGVNu6kKDegFmlyXcRdYhWPkydLgeut1hdjUOIUEpLsn8pFSeXbibkEAL74/pxahusWaXJNyJ3QaLfw/+wXDDvzz6umRlEpSiVqw2O68s3c/Mn45yRZtI3rq7h3T/Eb+18X1j9M0t06F+E7OrcRgJSlGjc0VlTPpsO2sPZTFhYBzPXtcJ/wsXzBbizG5Y8SJ0uA4S7jS7GoeSoBTVOpiez8Mfb+X0uRLeuD2BOxNbmF2ScEdlRcZd7noRcNM7XnPK/TMJSlGlHw9m8vu52wgOsPDZxP70bhVhdknCXX3/LGQdgPsWQmiU2dU4nASluKi5m47z/OJk2jeuz8zxiTLSRvxWfjqU5hsLgm2dCVdMgjZXmV2VU0hQit+Y9dNRXvhqL1d2iOade3sRFiT/TUQltnJjlvJ1bwHa2BbbHa563syqnEo+AeJXvk8+w4tf72VE58a8P6aX3LQRv7CWwd7FsP5t48ZNz7EQP8y4HtnmKo9Z1uFySFCK83alnuMP87aT0KwB/7m7p4SkMGhtdPv56W1jqrTItnD7R9D1VrMrcxkJSgFAfkk5j3+6ncjQIGbc34d6gTITuQCspbBkEuz6HOKHwuh3oc1wt15a1hmcerRKqZFKqQNKqRSl1NMXeb6BUuorpdROpVSyUspz52HycFMWJ5N6toi37+5BdH2ZbFdgzEr+yS1GSF41GcYthnYjfC4kwYktSqWUBXgPGAGkAluUUku01nsr7fZ7YK/W+kalVDRwQCk1V2td5qy6xG8t2n6KBdtP8cTV7UiMa2R2OcId5J+B2TfB2aNw24fQ7XazKzKVM0+9+wIpWusjAEqpecBooHJQaqC+MpbgCwNyAKsTaxIXOJFdxORFe0hsFcHjV7Y1uxzhDvLPwKwbIC/N6BcZN8jsikznzDZ0M+BkpcepFdsqexfoBKQBu4E/aq3tCJcotdqYNG87SsFbd/eQmzcC8k7DrOsh/zSM/VJCsoIzPxkXG8OkL3h8LbADaAr0AN5VSoX/5oWUmqiU2qqU2pqZmenoOn3Wy1/vY+fJc/zj9gSZS1JASS7MvsHoSD52AbQaYHZFbsOZQZkKVB4Y3Byj5VjZBGCBNqQAR4GOF76Q1nq61jpRa50YHR3ttIJ9ycLtqXyy8TgTh7RmZFeZKk0A3/4Nco7AvZ9Dy35mV+NWnHmNcgvQTikVD5wC7gbuvWCfE8BwYK1SqjHQATjixJp82pHMAlbuz2DLsRxWHcikb3wj/nptB7PLEu7gwLewYw4M/gvEDTS7GrfjtKDUWluVUo8D3wEWYKbWOlkp9WjF89OAl4BZSqndGKfqT2mts5xVky/bdzqPW6b+REm5nVaRIdzcoylPXttRrksKSN0KCyZCTBcY+pTZ1bglp3Y411ovBZZesG1ape/TgGucWYOA3OJyHp2TRHhwAN8/cQUtI+V6pKhw7Cf49E4IjTZOuf2lD+3FSHPCy2mt+b8vdnLqbDFTx/SSkBS/SJoFn9wM9WNhwjJoKHONVkWGMHq5aT8e4Ye96Tx3Q2fpTC4M1lJY9lcjKNtcZXQoD5H/G9WRoPRi6w5l8Y/v9nN9QiwPDIwzuxxhlvISSNsGGXshYx8cWweZ+2HQn42hiX4yrr8mEpReKiUjn9/NTaJtTBiv35aA8rKp+UUtlRfDzGvh9E7jcVA4xHSCOz+BzjeZW5sHkaD0QtkFpTwwaytB/n58eH8fmXjXV2kN3/zFCMkb34a2V0N4M69bz8YV5BPkZcptdh75JIn0vBLmTexPi0Zy88ZnbZ0JO+YaXX56jze7Go8mQellFm47xdbjZ/n3Xd3p2VIWA/NZu+fD0v+DtiOkb6QDSPcgL1Jus/PuqhS6NWvAzT0unH9E+Ix9XxkdyFsOgDs/lps1DiAtSi+ycPspTuQUMWNcoty88TXWMjj0HeycBweWQbPeRgfyQLn04ggSlF7CarPz3qoUujYLZ3inGLPLEa5it8G6f8OGd6H4LITGwIDHYMiTEFTf7Oq8hgSll/giKZXj2UV8IK1J31GQCfMnwLG10H4U9HkQWl8JFvlYO5r8jXqB3OJy3vzuAH3iIrhaWpO+oSQP5twKWYdg9FToca90+3EiCUov8PbyQ+QUlTH7xr7SmvQF1jL4fCykJxvXIduNMLsirydB6eFWH8jgo/VHuadvS7o2a2B2OcLZtDaWjz36I9z8voSki0j3IA+2K/Uckz7dTofG9Xnu+s5mlyNcYdUrsGseXPmscbotXEJalB5Ia82Cbad44atkGoYGMHN8H+oFSl85r3doOax5A3qONe5qC5eRoPQw54rK+OO8Hfx4MJPEVhG8dXcPmjasZ3ZZwhU2Tzfmjrz+33LjxsUkKD3IyZwi7p+5mdSzxbx4Uxfu698KPz/5wPiEvDRI+QEG/Qn8A82uxudIUHqQ5xfvIbOglLkP96OPTMLrW3bMBW03TruFy8nNHA+x/0weqw5k8vDg1hKSviZ5Eax/B+IGQ6PWZlfjk6RF6SH+++MRQgItjBvQyuxShKuU5MK3zxityaa94KZ3zK7IZ0lQeoAtx3JYsjON+wfE0TBErk95vbw02Pi+saZNWQEM+SsM/StYAsyuzGdJULq5kzlF/H7uNlpE1OOJEe3MLkc4U+ZBY4KL3V+AtkHnm2HQExDb3ezKfJ4EpRvLzC/lvg83UVJu45MH+xEeLC0Kr2QthbX/NL4sgcbkFv1/BxFxZlcmKkhQuqnc4nLGzdxMel4pcx7qR4cmMmWWV9IaPrsbDq+EhLvg2lcgNMrsqsQFJCjdkNaaiR9vJSUjnw/v70PvVrKkg9c6vNL4GvESDPyD2dWIKkj3IDd0ID2fTUdzeHpUJ4a0jza7HOEsWsPKl6FBC+j3iNnViGpIULqhVfszAbghIdbkSoRTHVgGaduMO9r+QWZXI6ohQemGVh3IoHNsOI3Dg80uRTiL3Q6r/m50IO8uswC5OwlKN5NbVE7S8bNc1VFmKvdqexdB+h4Y9jdZusEDSFC6mYXbU7HZNVdKUHqv/UvhqycgpjN0vdXsakQtyK8yN3Eiu4j/93Uyy/dl0K1ZA3q0aGh2ScKRclPh5GY4lWSsmBjbQ9bc9iASlG5g9vpj/H3pPvz9FM+M6siEgfFYZPo073H2OMwYDoXGTTp6j4eRr0OAXIP2FBKUJvt8ywmmLElmeMcY/n5LN5o0kA+PVynJhU/vAlsZjP/G6AoUIRObeBoJShNtOpLNMwt2M7hdFO+P7U2gv1wy9io2K3wxHrIPwdgFEDfI7IrEZZKgNNGCbacIC/JnmoSkd/rhOWPUzU3vQOuhZlcj6kA+nSY5nl3IygMZ9G8dSWiQ/L7yOnsWwMap0O9R6DXO7GpEHckn1ARrDmby2Nxt+Cl4eIjMWO110pNh8ePQop8xhlt4PAlKE0xdnULDkADmTexP84gQs8sRjnQqCebcBkH14Y5ZshCYl5BTbxOkZBQwsE2UhKS3ObYOZo+GoHB44FsIb2p2RcJBnBqUSqmRSqkDSqkUpdTTVewzTCm1QymVrJT60Zn1uIOcwjKyCspo1zjM7FKEI+341GhJNmhmhGSjeLMrEg7ktFNvpZQFeA8YAaQCW5RSS7TWeyvt0xCYCozUWp9QSnn9uL1D6fkAtGssE/F6hfIS+PYpY32buMFwx2wIjTS7KuFgzrxG2RdI0VofAVBKzQNGA3sr7XMvsEBrfQJAa53hxHpMV1xmY+ZPRwHoIEHp+axl8NldcGQ1DPoTXDlZJrjwUs78V20GnKz0OBXod8E+7YEApdRqoD7wttb6YyfWZJoT2UVM/GQrB9LzeXpURxmB4+m0hq/+aITkTe9Cr/vMrkg4kTOD8mKDlfVF3r83MByoB2xQSm3UWh/81QspNRGYCNCyZUsnlOpcG49k88gnSWit+Wh8H4Z18PorDN5v9Wuw81MY+rSEpA9w5s2cVKBFpcfNgbSL7POt1rpQa50FrAF+szan1nq61jpRa50YHe1ZSyMUlVl5Yt4OIkMD+WrSIAlJb7D+XfjxNegxBoZd9B6l8DLODMotQDulVLxSKhC4G1hywT6LgcFKKX+lVAjGqfk+J9bkcu+vPsyZvBLeuD2BVpGhZpcj6mrzB/D9s9B5NNz4H1Ayy5MvcNqpt9baqpR6HPgOsAAztdbJSqlHK56fprXep5T6FtgF2IEZWus9zqrJ1U7mFPHfNUcY3aMpiXGNzC5H1NW2T2Dp/0GH6+C2D+XGjQ9x6r+01nopsPSCbdMuePwP4B/OrMMsry3bj0Upnh7V0exSRF3tXQJLJkHbq40RN5YAsysSLiQjc5wkp7CMb5PPMO6KVsQ2qGd2OaIuyopg2V8htjvcNUdWTPRBcu7gJMv3pmOza25MkGFsHm/jVMg/Dbd/BAHyS88XSYvSSZbtOU3ziHp0aRpudimiLtKT4ae3jeuSrQaYXY0wibQoHUxrzcLtp/gpJZtxA1qh5K6o5ynKgRMbYNN/4eiPEBgGV79gdlXCRBKUDjZn43GeW5xM59hwxg+MM7scURt5p+Gnt+DMHsg68MsiYOHNYPgU6HW/jN/2cRKUDlRmtTN19WESW0Xwv0cG4CcrKbq/o2tg/gNQmg9NEqD9SIjuYKy5HT9E7m4LQILSoT7ecIzTuSW8dluChKQn2DDV6Dwe2dZYITG6g9kVCTclQekAWmumrj7MP747wOB2UQxpF2V2SaI6WsOPb8DqV6DTjXDz+8aM5EJUQYLSAV5dtp/pFSNw3rg9QW7guLOCDFj5MmybbYzVvukd8LOYXZVwcxKUdVRmtfPxhmPc2L0pb93VQ0LSXdmsRgty4/tgLYUBjxsLf/lJDzlRMwnKOlp/OIuScjs3JMRKSLqzTdNg7T+h620w7G8Q1dbsioQHkaCsg5zCMv62YDfNI+oxqK1cl3RbBRnw4+vQdgTcPtPsaoQHuuTzDqWURSk1xhnFeJrnFu8hq7CM98f0JjRIfue4rZUvQXkRjHzV7EqEh6oyKJVS4UqpZ5RS7yqlrlGGScAR4E7Xlei+dpw4x/XdYunWvIHZpYiqpO0wpkfr9yhEtTO7GuGhqmsGfQKcBTYADwFPAoHAaK31DueX5t601mTml9I4XNa+cVtaw/eTIaQRDP2r2dUID1ZdULbWWncDUErNALKAllrrfJdU5ubOFZVTZrMTXV+m3HJbKSvg2FoY9QYES6tfXL7qrlGW//yN1toGHJWQ/EVGfikAMRKU7slug+VTICIOek8wuxrh4aprUXZXSuXxy2qK9So91lprn54/7GhWASBB6bZ2fQ7pe4wlG/wDza5GeLgqg1JrLcMVqnAwPZ+nvjS6BXVpJqd0bqesCFa8BM16G/0mhaijKoNSKRUMPAq0xVj8a6bW2uqqwtzZf1YcAuCzh/sTJt2C3M/G9yA/DW7/UFZJFA5R3TXK2UAisBu4DvinSypyc+U2Oz8ezOSazo1p0SjE7HLEhQoyYN1b0PEGaHWF2dUIL1Fdc6hzpbveHwKbXVOSe9tyNIf8EitXd25sdiniYla/CuXFMiO5cKja3vWWU+4Ky/dlEOjvx2CZSs39nNkNSbOgz0PSuVw4VHUtyh4Vd7nBuNPt83e9tdYs35fOFW0iCQmUa5NuRWtY9jQEN4QrnzG7GuFlqmtR7tRah1d81dda+1f63udCEiAlo4ATOUUM7ySn3W5n72I4vg6umgz1IsyuRniZ6oJSu6wKD7Fg+yn8FFwj1yfdS2m+MVSxcVfoPd7saoQXqu78MUYp9eeqntRa/8sJ9bitMqudL5NSubJDjIzvdjcrXoLcVKNzucxWLpyguqC0AGH8MjLHp32+9SQZ+aWMuyLO7FJEZSc2webp0HcitOxndjXCS1UXlKe11v/PZZW4sZJyG++uPESfuAhZOMydlBfDksehQXMY/rzZ1QgvVl1QSkuywpyNx0nPK+Xtu3vKcg/uoLTA6Aa0oWIEzpgvISjM7KqEF6suKIe7rAo3VlBq5f3VhxncLor+rSPNLsf32G3w+X2QsrzSNitoG8QNhlveh9bDTCtP+IbqJsXIcWUh7mr6miNkF5bxf9d0MLsU37ThPTjwDfQYC6EVlz38LNB+FLToY25twmdIr+lqZOSXMGPtEa5PiKV7i4Zml+N70vcaa3B3vAFGvysTXAjTyKLG1Zi66jBlVjtPSmvS9U5ugdk3QnA43PCWhKQwlQRlNbYez+GKtlHERYWaXYpvSV4Is2+AoPowYRmERZtdkfBxEpTVyMgrpUm4zGDuMnY7rPkHfDEeYrvDQytkcgvhFuQaZRVsdk1WQSkx9WUUjkuU5MGi38H+r6HbnXDTOxAgf/fCPUhQViG7sBS7hhhpUTpfxn74332QfRiufRX6/06uSQq3IkFZhYw8Y5XF6DAJSqc5dxLWvAHb5xoz/oxbDPGDza5KiN+QoKzCsexCAFnuwdFKco1uP8kLIekjY1vfh2HwXyAsxtzahKiCBGUVktPy8PdTtGssQ+PqrDDbWGP76I9w7oSxTVmg51gY+ldjrLYQbsypQamUGgm8jTET0Qyt9WtV7NcH2AjcpbWe78yaamvPqVzaN65PkL9M21UnB76FJZOg+Cx0vB563Q9NEqBpD2lBCo/htKBUSlmA94ARQCqwRSm1RGu99yL7vQ5856xaLpXWmr1peQzvJB/ky1aaD9/9DbZ9bEyoe99CaNLV7KqEuCzObFH2BVK01kcAlFLzgNHA3gv2mwR8CbjNwN0zeSVkF5bRpWkDs0vxTOXF8PHNkLYNBv0Jhj0D/nJTTHguZwZlM+BkpcepwK9mVlVKNQNuAa7CjYJyzyljTbWuzXxyaaC6sdth4SNwKgnu/Bg632R2RULUmTNH5lysI9yF6/C8BTyltbZV+0JKTVRKbVVKbc3MzHRUfVVKTstFKejYRILykq18yVjo65qXJCSF13BmizIVaFHpcXMg7YJ9EoF5FZPhRgHXKaWsWutFlXfSWk8HpgMkJiY6fdGzPadyiY8KJTRIOgVcki0zYN2/jAW+BjxudjVCOIwzk2AL0E4pFQ+cAu4G7q28g9Y6/ufvlVKzgK8vDElXs9k1W46dZWSXJmaW4Xm2z4Fv/gLtR8J1b8rIGuFVnBaUWmurUupxjLvZFmCm1jpZKfVoxfPTnPXedbHvdB65xeVc0VZmM6+13fNh8ePQ5iq4YzZYAsyuSAiHcuq5pdZ6KbD0gm0XDUit9Xhn1lJb6w9nATBAln2onUM/GDdvWg2Eu+bKRBbCK8k0axdYl5JNm+hQYmTt7uppDQeWwf/GQUxnuOczCJThnsI7yd2KSorKrGw8ks19/VuZXYp7y0uDuXdA+h5o1AbGzDdmIhfCS0lQVrLhcDZlVjtXdpAROVUqPgtzbjPGbN/8PnS9TTqTC68nQVnJyv0ZhARa6BMfYXYp7klrmP8gZB2CMV9AmyvNrkgIl5CgrKC1ZvWBTAa2jZKJMKqSsgIOrzAm15WQFD5EbuZUOJRRwKlzxVzVUU67L8puhxUvQMOW0Ochs6sRwqWkRVlh5f4MAIZ1kBX/Lip5AZzZDbdMB/9As6sRwqWkRVlh1f4MOjapT2yDemaX4n6sZbDyZWO6tG53mF2NEC4nQQlYbXa2nzjH4HZRZpfifsqK4Ptn4exRGP48+Ml/GeF75NQbSDtXQpnNTtsYWfbhPK1h31fG5Lu5J42ZydtdY3ZVQphCghI4klUAQHyUBOV5ix+HHXOMUTfjv4G4QWZXJIRpJCiBY1nGiotxUTIED4D93xghOeBxuPpFsMh/E+Hb5IITcCy7iNBAi6zhDcZaN0ufhJgucPULEpJCIC1KAA5nFhAfHYqSORSNu9t5aTJdmhCV+HyLUmtNcloenWNlUgdSk2DTf6HPg9DCbZYwEsJ0Ph+Up3NLyCkso2szH19xsawIvvoDhDU2ugEJIc7z+VPv5DRjxcUuTX24RWm3wYKHIT0Z7pkHwT7+S0OIC/h8UO48eQ6Ln6KTL596f/8c7P/amOyiw0izqxHC7fj8qffW4zl0aRpOSKCP/s7Y9zVsfA/6PQoDHjO7GiHckk8HZZnVzo6T5+jdykfnnywvMUbeRHeCa/5udjVCuC0fbUYZUjIKKCm307Oljwbl2n/CueMwbrH0lxSiGj7doswsKAWgaQMfXEhs/Tuw5g1IuAtaDzO7GiHcmk8HZXZFUEb62oicn/4D30+GzjfD6PfMrkYIt+fT51vZBWUARIb50ES0KSvgh+eNkLztQznlFqIWfPpTklVYSqDFj/pBPvDXUJQDuz6HH1+HmE5w81QJSSFqyac/KdkFZUSGBXr3GO/8dOM0e+9isJVCs0S4dToEhppdmRAew6eDMjO/1PtPu7/6IxxZBb3GQe/7oUk3sysSwuP4dFCm55XQPMKL56A8sAwOLoMRL8HAP5hdjRAey6fvep/JK6FJAy+9411eDMueguiO0P93ZlcjhEfz2RZlSbmNc0XlNAn30j6U6/5tdCa//2uZV1KIOvLZFmV6XgkAjb0xKLMPw7q3jKVl4webXY0QHs9ng/JMrhGUTbxtVE7adlgwESyBxrVJIUSd+eyp95mKFqVXnHpby+DwCmN28iOrICgcbvg3hMeaXZkQXsFngzKrYlROTH0PDsojP8LuL2DfEijJhdAYY0GwxAdk8l0hHMhngzK3uByloH6wh/4V7FkA8ydAYH3odAN0vc2Y3EJu3AjhcB6aEnWXV1xO/SB//Pw8dFTOlg8hIh4e2wAB9cyuRgiv5rM3c/KKywmv56Gtr+zDcHwd9BwrISmEC/hsUJ7OLSG6vod2Nt8+B5Qf9LjX7EqE8Ak+G5SHMgpoFxNmdhmXzmaFHZ9C2xEQ3tTsaoTwCT4ZlLlF5WQVlNLWE4MyZTkUnIFe95ldiRA+w6lBqZQaqZQ6oJRKUUo9fZHnxyildlV8rVdKdXdmPT9LycwH8Myg3P4JhEZDe1lWVghXcVpQKqUswHvAKKAzcI9SqvMFux0FhmqtE4CXgOnOqqeyQ+kFALSLqe+Kt3Ocggw4+C10v0e6AQnhQs5sUfYFUrTWR7TWZcA8YHTlHbTW67XWZysebgSaO7Ge81IyCggO8KNZQw+7Y7zzM7BboaecdgvhSs4MymbAyUqPUyu2VeVBYJkT6zkvJbOA1lFhntWH8sweY6KLlgMgur3Z1QjhU5zZ4fxiKaQvuqNSV2IE5aAqnp8ITARo2bJlnQs7lF5A71YetJb3md0w+yajz6SsmiiEyzmzRZkKtKj0uDmQduFOSqkEYAYwWmudfbEX0lpP11onaq0To6Oj61RUfkk5p84V06GJh1yfPL0LZt8IASEw/muIbGN2RUL4HGcG5RagnVIqXikVCNwNLKm8g1KqJbAAuE9rfdCJtZx3MN24490p1gOCct/XMPsGCAwzQrJRa7MrEsInOe3UW2ttVUo9DnwHWICZWutkpdSjFc9PA54HIoGpFSshWrXWic6qCWDfaSMoOzQJd+bb1E1JHiyfAltnQmwPuPNjiGhldlVC+CynToqhtV4KLL1g27RK3z8EPOTMGi60/0we9YP9aequE/Ye/B6+/hPknYIBj8Pw58HfQ4daCuElfG72oP2n8+nUJNy91vK2lhmrJW6fA4e+NxYEe/B7aNHX7MqEEPhYUGqtOXAmn5t7VtdLyYVKciFpFmycBvlpENYErpxsLC0rrUgh3IZPBeWpc8Xkl1rpaPaNnPx0WP8fSJoNZfkQPwRufAvaXg1+FnNrE0L8hk8F5f6KGzkdzbyRc+4EfHQd5KVBl1vgiknQtId59QghauRTQbnvdB5KYV4fyrw0o09kaR48vAKa9jSnDuG1ysvLSU1NpaSkxOxS3FZwcDDNmzcnIKD28yX4VFDuScslPjKUsCATDjv/jDG6pjAbxi2WkBROkZqaSv369YmLi3OvG5ZuQmtNdnY2qampxMfH1/rnfGo+yj2n8ujSzITVCXNT4aNRkH8axs6H5r1dX4PwCSUlJURGRkpIVkEpRWRk5CW3uH0mKM8VlXHqXDFdmrr4+mTuKSMkC7PhvkXQsr9r31/4HAnJ6l3O34/PBGVmfimAa6dWKz4Lc26DorMwbhG06OO69xbCw8XFxZGVlVXnfRzBZ65RFpfbAAgJdFH3m/Ji+OweyDkMY7+EZr1c875CCIfzmRZlcZkRlPUCXBCUNivMfwBObIRbpxv9JIXwAceOHaNjx4489NBDdO3alTFjxrB8+XIGDhxIu3bt2Lx5Mzk5Odx8880kJCTQv39/du3aBUB2djbXXHMNPXv25JFHHkHrX2ZlnDNnDn379qVHjx488sgj2Gw2lx6Xz7Uog13RovzhOTiwFK570+grKYQJXvwqmb1peQ59zc5Nw5lyY5dq90lJSeGLL75g+vTp9OnTh08//ZR169axZMkSXnnlFVq0aEHPnj1ZtGgRK1euZNy4cezYsYMXX3yRQYMG8fzzz/PNN98wfbqxMsy+ffv4/PPP+emnnwgICOCxxx5j7ty5jBs3zqHHVh2fCcqSche1KFNWwMap0PcR6Puwc99LCDcUHx9Pt27dAOjSpQvDhw9HKUW3bt04duwYx48f58svvwTgqquuIjs7m9zcXNasWcOCBQsAuP7664mIMCbXXrFiBUlJSfTpY1zjLy4uJiYmxqXH5DNBWeyKoCzKgcW/Nya1GPGi895HiFqoqeXnLEFBv8xT4Ofnd/6xn58fVqsVf//fxs7Pd6Ivdkdaa83999/Pq6++6qSKa+Yz1yjziq0AhAQ5MSiXPgmFWcZ1yQAPW7hMCBcZMmQIc+fOBWD16tVERUURHh7+q+3Lli3j7Flj3cHhw4czf/58MjIyAMjJyeH48eMurdlnWpSpZ4sI8vcjKtRJs/Kc2gZ75sOQv0KsS5YnF8IjvfDCC0yYMIGEhARCQkKYPXs2AFOmTOGee+6hV69eDB069Pz6WJ07d+bll1/mmmuuwW63ExAQwHvvvUerVq6bzFpVvrPkCRITE/XWrVtrvf/IL0fSK6YXOcdu41BGPiv+Msw5hc25zQjLP+6EYDeePV14tX379tGpUyezy3B7F/t7UkolVbXCgs+cep88W0SLRiHOefHjGyBlOQx6QkJSCC/kM0GZkV9Kk3AnLP9gs8IPz0NoDPSRu9xCeCOfuUZZVGp1zqxBK16E1M1w6wcQ6KQWqxDCVD7RotRoCstshDg6KPcuMWYqT3wQEu507GsLIdyGTwSlzW7csApzZNegrBRY9Bg06w0jzevfJYRwPp8ISmtFUIYEOqhFqTUseBgsAXDHbFkITAgv5xNBWWa1AxBer/ZTv1fr4LeQtg2ueQkatnDMawohfsNV06jVxCeCsqjMGJXjkLkotYbVr0FEHCTcXffXE8JLaa2x2+1ml+EQPhKUxjhvhwTlwe/g9A4Y8iRYfKbTgBC1cuzYMTp16sRjjz1Gr169ePDBB0lMTKRLly5MmTLl/H5xcXFMmTKFXr160a1bN/bv3w9UP9Xav/71L7p27UrXrl156623zr9fTdO6OYJPfNJ/PvWOCK3jqbe1DFa/Cg1bQcJdDqhMCCda9jSc2e3Y12zSDUa9Vu0uBw4c4KOPPmLq1Knk5OTQqFEjbDYbw4cPZ9euXSQkJAAQFRXFtm3bmDp1Km+++SYzZsyocqq1pKQkPvroIzZt2oTWmn79+jF06FAiIiJqnNZt0aJFdT5sn2hRllnthAX5E+Rfh7veZUXw+RijNTn8eeNGjhDiN1q1akX//sbaUP/73//o1asXPXv2JDk5mb17957f79ZbbwWgd+/eHDt2DIA1a9YwduxY4NdTra1bt45bbrmF0NBQwsLCuPXWW1m7di3wy7Rufn5+F53WzRF8okVZYrXTKDSwDi+QB5/dDcfXw41vQ7fbHVecEM5SQ8vPWUJDQwE4evQob775Jlu2bCEiIoLx48f/avXDn6dfs1gsWK3W89urmmqtKjVN6+YIPtGiLCq1Xvr1SZsVkmbD5/fB2wlwchPcNgN6j3dKjUJ4m7y8PEJDQ2nQoAHp6eksW7asxp+paqq1IUOGsGjRIoqKiigsLGThwoUMHjzYqfVX5hMtysIyGy1iLjEol0+BDe9CeHPocB30GANxA51ToBBeqHv37vTs2ZMuXbrQunVrBg6s+fNT1VRrvXr1Yvz48fTt2xeAhx56iJ49ezrs1LomXj/N2rXzR3L8VAy/7/ock4a3q90PJS+CL+43Jrm47h8g6yQLDyHTrNWOTLN2gZ+HLzaLqGWLMuuQsZxDs0S49hUJSSGE7wRl49pMsXbuBMy93RiSeOds8K/DDSAhhNfw+muUtopLC9H1axiPnXMEZt8EpXkwdiE0aO6C6oQQnsDrg/LnS7DVrr5oLYXP7oHyIrj/a4hNcE1xQgiP4ANBaSRlUEAVVxm0NkbbZO6HMV9KSAohfsPrg9Jm1/j5qd+uvmgtg91fGF2AMvZC93ug3dXmFCmEcGtefzPHatcEB1jw86u4e12UA+v+bXQiX/wYoODmaXDTO6bWKYS3uOKKKy66ffz48cyfP/+yXvOFF17gzTffrEtZdeITLcqQAAtk7IdN78POz8FaDPFDYfS70Ga4dAESwoHWr19vdgkO5/1BqTUNLcUwbRD4WaDbHdDvEWMWFCGEw4WFhVFQUIDWmkmTJrFy5Uri4+N/NV47KSmJP//5zxQUFBAVFcWsWbOIjY3lgw8+YPr06ZSVldG2bVs++eQTQkLMX7TPqUGplBoJvA1YgBla69cueF5VPH8dUASM11pvc2gRGpqWHQc/f/jDNghv6tCXF8Jdvb75dfbn7Hfoa3Zs1JGn+j5Vq30XLlzIgQMH2L17N+np6XTu3JkHHniA8vJyJk2axOLFi4mOjubzzz/n2WefZebMmdx66608/LCx7PPkyZP58MMPmTRpkkOP4XI4LSiVUhbgPWAEkApsUUot0VrvrbTbKKBdxVc/4P2KPx3GDxuNytOg5xgJSSFcaM2aNdxzzz1YLBaaNm3KVVddBRjzVe7Zs4cRI0YAYLPZiI2NBWDPnj1MnjyZc+fOUVBQwLXXXmta/ZU5s0XZF0jRWh8BUErNA0YDlYNyNPCxNtrkG5VSDZVSsVrr044qIpRilNbQ/zFHvaQQHqG2LT9nqmrKtC5durBhw4bfPDd+/HgWLVpE9+7dmTVrFqtXr3ZBlTVz5l3vZsDJSo9TK7Zd6j4opSYqpbYqpbZmZmbWvgJbObfn59KSVhDZpvY/J4SosyFDhjBv3jxsNhunT59m1apVAHTo0IHMzMzzQVleXk5ycjIA+fn5xMbGUl5efn66NXfgzBblxW4lXzhVUW32QWs9HZgOxuxBta7AEsCN139FiMVW6x8RQjjGLbfcwsqVK+nWrRvt27dn6NChAAQGBjJ//nz+8Ic/kJubi9Vq5YknnqBLly689NJL9OvXj1atWtGtWzfy8/NNPgqD06ZZU0oNAF7QWl9b8fgZAK31q5X2+S+wWmv9WcXjA8Cw6k69L3WaNSF8iUyzVjvuNM3aFqCdUipeKRUI3A0suWCfJcA4ZegP5Dry+qQQQjiC0069tdZWpdTjwHcY3YNmaq2TlVKPVjw/DViK0TUoBaN70ARn1SOEEJfLqf0otdZLMcKw8rZplb7XwO+dWYMQQtSV14/1FsLXeNryLq52OX8/EpRCeJHg4GCys7MlLKugtSY7O5vg4FqseFCJ14/1FsKXNG/enNTUVC6pv7GPCQ4OpnnzS1vBQIJSCC8SEBBAfHy82WV4HTn1FkKIGkhQCiFEDSQohRCiBk4bwugsSqlM4Pgl/lgUkOWEcszgLcfiLccBcizu6lKPpZXWOvpiT3hcUF4OpdTWqsZwehpvORZvOQ6QY3FXjjwWOfUWQogaSFAKIUQNfCUop5tdgAN5y7F4y3GAHIu7ctix+MQ1SiGEqAtfaVEKIcRl86qgVEqNVEodUEqlKKWevsjzSin1n4rndymleplRZ01qcRxjKurfpZRar5TqbkadtVHTsVTar49SyqaUut2V9V2K2hyLUmqYUmqHUipZKfWjq2usjVr8/2qglPpKKbWz4jjcdp5YpdRMpVSGUmpPFc875jOvtfaKL4zJgQ8DrYFAYCfQ+YJ9rgOWYazV0x/YZHbdl3kcVwARFd+PcsfjqO2xVNpvJcbcpbebXXcd/l0aYqwy2rLicYzZdV/mcfwNeL3i+2ggBwg0u/YqjmcI0AvYU8XzDvnMe1OL8vzyuFrrMuDn5XErO788rtZ6I9BQKRXr6kJrUONxaK3Xa63PVjzcCFzaVCiuU5t/E4BJwJdAhiuLu0S1OZZ7gQVa6xMAWmt3PJ7aHIcG6itjrdkwjKC0urbM2tFar8GoryoO+cx7U1A6bHlck11qjQ9i/MZ0RzUei1KqGXALMA33Vpt/l/ZAhFJqtVIqSSk1zmXV1V5tjuNdoBOQBuwG/qi1trumPIdzyGfem6ZZc9jyuCardY1KqSsxgnKQUyu6fLU5lreAp7TWNqMB47Zqcyz+QG9gOFAP2KCU2qi1Pujs4i5BbY7jWmAHcBXQBvhBKbVWa53n5NqcwSGfeW8KylSgRaXHzTF+I17qPmarVY1KqQRgBjBKa53totouVW2OJRGYVxGSUcB1Simr1nqRSyqsvdr+/8rSWhcChUqpNUB3wJ2CsjbHMQF4TRsX+VKUUkeBjsBm15ToUI75zJt9MdaBF3X9gSNAPL9cpO5ywT7X8+sLu5vNrvsyj6MlxsqVV5hdb12P5YL9Z+G+N3Nq8+/SCVhRsW8IsAfoanbtl3Ec7wMvVHzfGDgFRJldezXHFEfVN3Mc8pn3mhal9pLlcWt5HM8DkcDUipaYVbvhRAa1PBaPUJtj0VrvU0p9C+wC7MAMrfVFu62YpZb/Ji8Bs5RSuzEC5imttVvOKKSU+gwYBkQppVKBKUAAOPYzLyNzhBCiBt5011sIIZxCglIIIWogQSmEEDWQoBRCiBpIUAohRA0kKIXHqphtaEelr7iK2XtylVLblVL7lFJTKvatvH2/UupNs+sXnsNr+lEKn1Sste5ReYNSKg5Yq7W+QSkVCuxQSn1d8fTP2+sB25VSC7XWP7m2ZOGJpEUpvJY2hhImYYxXrry9GGMss7tNiCLclASl8GT1Kp12L7zwSaVUJMawteQLtkcA7YA1rilTeDo59Rae7Den3hUGK6W2YwwjfK1iiN6wiu27gA4V28+4rFLh0SQohTdaq7W+oartSqn2wLqKa5Q7XFyb8EBy6i18jjbmh3wVeMrsWoRnkKAUvmoaMEQpFW92IcL9yexBQghRA2lRCiFEDSQohRCiBhKUQghRAwlKIYSogQSlEELUQIJSCCFqIEEphBA1kKAUQoga/H8e7aAyZiMWWwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 360x360 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.figure(figsize=(5,5))\n",
"plt.plot(df_scores.fpr, df_scores.tpr, label=\"model\")\n",
"plt.plot(df_rand.fpr, df_rand.tpr, label=\"random\")\n",
"plt.plot(df_ideal.fpr, df_ideal.tpr, label=\"ideal\")\n",
"plt.xlabel(\"FPR\")\n",
"plt.ylabel(\"TPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "936e4a82-9ae3-4402-a065-9968e83ca572",
"metadata": {},
"source": [
"* We want to get as close to the ideal model\n",
"* To see how goo a model is we can draw the following plot:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "bcc7efec-0378-4be2-9731-772eb93bd0a7",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAE9CAYAAABtDit8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1XklEQVR4nO3dd3hUZd7G8e+TThqEFDqE3nsoigXFQm+igmhCLKxr2XWLa1nXsq+rrrq7rivooot0UKQ33VVRFEEh9E7ooaRCejKZzPP+cUaNmJBAZnJm5vw+15WLzMxJcp/A3Jz6PEprjRBCiKr5mR1ACCE8nRSlEEJUQ4pSCCGqIUUphBDVkKIUQohqSFEKIUQ1AswOcLliYmJ0fHy82TGEED4mJSUlS2sdW9lrXleU8fHxbN261ewYQggfo5Q6UdVrsusthBDVkKIUQohqSFEKIUQ1vO4YZWXKyspIS0ujpKTE7CgeLSQkhObNmxMYGGh2FCG8ik8UZVpaGhEREcTHx6OUMjuOR9Jak52dTVpaGq1btzY7jhBexSd2vUtKSoiOjpaSvASlFNHR0bLVLcQV8ImiBKQka0B+R0JcGbcVpVJqplIqQym1p4rXlVLqTaVUqlJql1Kqj7uyeJv4+HiysrJqvYwQwjXcuUU5Cxh6ideHAe2dH1OBt92YRQghrpjbTuZorTcopeIvscgYYI42hljfrJRqoJRqorU+665M7nT8+HGGDh3KNddcw+bNm+nZsyfJyck899xzZGRkMH/+fNq1a8e9997L0aNHCQ0NZcaMGfTo0YPs7GwmTZpEZmYm/fv3p+Ko8/PmzePNN9/EZrMxYMAApk+fjr+/v4lrKoR7aa2xOzRl5Q7K7BpbucP43Plhs+sfPy93YLM7KCv/8bkyWyltj8yhyS2/oXF0fZdkMvOsdzPgVIXHac7nflaUSqmpGFudtGzZsk7CXYnU1FQWL17MjBkz6NevHwsWLODrr79m5cqVvPTSS7Ro0YLevXuzfPlyPv/8cxITE9mxYwcvvPAC11xzDc8++yxr1qxhxowZAOzfv58PPviAjRs3EhgYyEMPPcT8+fNJTEw0eU2FL9JaU2p3UFJWTnFZOaVlDkrtDkrt5cafZT9+bqvieeNxObZyR6Vf/9PS086Sq/DY+fmVzlATRBlvBb5Jb/8U9jbqQOMb73LJ78bMoqzszEKlvx6t9QxgBkBCQsIlf4UvrNrLvjN5tU9XQZemkTw3qmu1y7Vu3Zru3bsD0LVrV4YMGYJSiu7du3P8+HFOnDjBkiVLALjxxhvJzs4mNzeXDRs2sHTpUgBGjBhBVFQUAJ999hkpKSn069cPgOLiYuLi4ly6bsI3FJbaycgvJT2vhPS8EjLySskvtVNss1NcVk6xzUFxmZ1iW7nzsfPP7z93PnbUYgotpSAkwJ+gAD+CA/wIDvQjOMCf4AA/ggL8CPL3IyIkgOAAPwL9f/wIClA/fezvfBzw/esVnqvya/wIopQW//sFoSdSyLvxZdpdfafLfr9mFmUa0KLC4+bAGZOyuERwcPAPn/v5+f3w2M/PD7vdTkDAz3/d35+JruyMtNaapKQkXn75ZTclFp5Oa01Gfiknc4qcJVhKhrMM0/NKSc83SrGg1F7p19cL9Cc0yJ+QQH/qBf34eVRYEE0D/annfP6HP7//PNBY7vuSCw7wdxbfT8vPKETjcYCfMu/KClsRLEqEE1/CqH8S2XeKS7+9mUW5EnhEKbUIGADkuuL4ZE22/Mxy3XXXMX/+fP70pz/xxRdfEBMTQ2Rk5A/PP/PMM6xbt47z588DMGTIEMaMGcNvfvMb4uLiyMnJIT8/n1atWpm8JsLVyh2atPNFpGYU/PBxOKOAI5kF5Jf8tASDAvxoFBlMo4gQOjeO5PoOwTSKDCEuwvizUWQwsREhRIYEWOOSsNICWDgRjn8NY6ZB78ku/xFuK0ql1EJgMBCjlEoDngMCAbTW7wBrgeFAKlAEJLsri6d4/vnnSU5OpkePHoSGhjJ79mwAnnvuOSZNmkSfPn24/vrrfzgO26VLF1588UVuueUWHA4HgYGBTJs2TYrSB1wosrH1+Hm2nMhh6/Hz7DmdS6nd8cPrsRHBtIsNZ2yvZrSLCyc+JozGzhKsXy/QGgVYEyV5MP92SNsC49+FHre75ccob5vXOyEhQV88HuX+/fvp3LmzSYm8i/yuzHO+0Ma09alsOJzJofQCAAL9Fd2b1ad3yyg6NAqnXVw47WIjqB8q9+NXq/gCzLsNzu6A2/4DXcfW6tsppVK01gmVveYT93oL4cm01nyUksbL6w6QW1zGNe1iGNOrGQmtoujZogEhgXK512UryoG5YyF9H9wxBzqNcOuPk6IUwo0OpefzzLI9fHc8h4RWUbw4rhudGkeaHcu7FWQaJZl1GCYugA63uP1HSlEK4QJaawpt5WTklZCZX0pGfik7Tl1g9jfHCQ8J4K+3def2vi3w85Nji7WSnw5zRsP5E3DXImh7Y538WClKIWrI4dCcyS3mSGYhR5xnpI9kFnDmglGOxWXlP/ua2/s256nhnWkYFmRCYh+TdwZmj4K8szB5MbS+ts5+tBSlEJXQWnMovYBP96dz4Fw+RzIKOJpVQEnZj2em69cLpG1sGL1aNCAuIpjYiGDiIoOJDQ8hznn5jpyUcZELp4ySLMyCe5ZCy4F1+uOlKIVw0lqz+3Qu6/ac4+M95ziWVQhAi4b1aBsbzlVto2kXF07b2HDaxobRMCxILtOpCznHYPZoKMmFxOXQvNIT024lRemhvp+WNyYmxuwoPqXUXk5WgY2s/FIy80vJLCglK7+UM7klbDiUyekLxfj7Ka5uG81917Tmlq6NiIsIMTu2dWUfMbYky4ogaQU07W1KDClKN9Bao7XGz89nxkX2GsW2clbtOsPC705y8Fz+T17TmkqPI4KxG53QKorHbmrPzV0a0SBUjimaLvOgsSXpKIOkVdC4u2lRpChd5Pjx4wwbNowbbriBTZs20atXL3bv3k1xcTETJkzghRdeAIwtxaSkJFatWkVZWRmLFy+mU6dOlxxq7e9//zszZ84E4P777+exxx6r0bBu/fv3N+V3YYZD6fks+PYkS7alkV9ip21sGBP7tcT/ov+rIkMCiXUeT4wJN/6MDg8iOECuZfQo6fuMs9somLIG4ky+SeL7rR9v+ejbt6++2L59+372XF07duyYVkrpTZs2aa21zs7O1lprbbfb9fXXX6937typtda6VatW+s0339Raaz1t2jR93333aa21fvTRR/ULL7ygtdZ69erVGtCZmZl669atulu3brqgoEDn5+frLl266G3btuljx45pf39/vWvXLl1eXq779Omjk5OTtcPh0MuXL9djxoypNKcn/K5cpdhm10tSTunbpm/UrZ5Yrds/vVb/auE2vflIlnY4HGbHE1fqzA6tX4nX+vWOWmceqrMfC2zVVfSO721RrnsSzu127fds3B2GvVLtYq1atWLgQONs3IcffsiMGTOw2+2cPXuWffv20aNHDwDGjx8PQN++fX8YXq2qoda+/vprxo0bR1hY2A9f+9VXXzF69Ohqh3XzVfklZby74ShzNp/gQlEZ8dGhPD28ExP6tpDLcLzd6RSYOw6CIiBpJUS3NTsRILveLvV9mR07dozXX3+dLVu2EBUVxZQpU34y++H3w6/5+/tjt/84MkxVQ61Vpbph3XxNqb2c+ZtP8tb6VHIKbdzatRGJV8VzVZtouZDbF5z6zrh3u16UcUwyynMGf/G9oqzBlp+75eXlERYWRv369UlPT2fdunUMHjz4kl9T1VBr1113HVOmTOHJJ59Ea82yZcuYO3duHayF53A4NCt2nuZv/z1E2vlirm4bzZPDOtGjeQOzowlXOb4RFtwB4Y2MLcn6zc1O9BO+V5QeoGfPnvTu3ZuuXbvSpk0bBg0aVO3XVDXUWp8+fZgyZcoPJ2buv/9+evfu7dO71hV9dTiTl9YeYP/ZPLo2jeSlcd25tn2MXL/oS45+AQsnGeWYtAoiGpud6GdkmDWL8Zbf1dHMAl5au59P92fQomE9fn9LR0b1aCq72L4m9VNYNBkatoHEFRBu3lQnMsya8Bq5xWX867PDzN50nOAAf54c1onkQfFy+Y4vOrgOPkyE2I5wzwoIizY7UZWkKIVH0FqzatdZXli5l5wiG3f0bcHvbu0gd8X4qn0r4aNk44qSu5dCaEOzE12SFKUw3dncYp5ZtofPDmTQs3l9Zt/bn27NXDMfs/BAe5bAkgegWV+4+yMI8fy/a58pSq21HOCvhqccj3Y4NGfzSjiRVcjOtFymrU/F7nDwzIjOJA9qjb8ch/RdOxfB8l9Ci4Ew+UMIjjA7UY34RFGGhISQnZ1NdHS0lGUVtNZkZ2cTElL3u7Kl9nI2H83h033pfHcsh+PZhT+ZSGtQu2heHteDltGhdZ5N1KFtc2Hlo8Y4kpMWQVCY2YlqzCeKsnnz5qSlpZGZmWl2FI8WEhJC8+Z1d31ayokc3vvqGBsOZVJoK6deoD8D2jTkug4xxMeE0To6jFYxYTStHyL/wfm6Lf+BNb+FtkNg4nwIrGd2osviE0UZGBhI69atzY4hKth0JJvkWd8RHhzImN7NuLlzI65qGy0TaVnR5nfg4yegw1C4fTYEet8JOp8oSuFZUk7kcN/sLbSICmXR1IFEhwdX/0XCN238J/zvWeg8Cm6bCQHeeS++FKVwqZ2nLjBl5hYaR4Yw/4EBUpJW9uVrsP5F6Doexs8Af++dFkOKUrjM3jO5JM78jgZhgcx/YIBcA2lVWsP6l2DDq9BjIoyZBv7eXTXenV54jD2nc5n83reEBfmz4P6BNKnvXQfrhYtoDZ8+DxvfgN53w6g3wc/7j0tLUYpa252Wy93/+Zbw4AAWTR1Ii4ZymY8laQ2fPA2bp0PCfTD8dfCR6VCkKEWtpJw4T/L73xEREiglaWUOB6x7HLa8BwN+CUNfBh+65EuKUlyRsnIHb32eylvrU2naIISFDwykeZSUpCU5HLD617BtDlz9K7j5zz5VkiBFKa5AakYBv/1wB7vSchnfuxnPje5K/Xree0ZT1IKjHFY8DDsXwnWPww1/9LmSBClKcZk+Sknjj8t2Exrkz9uT+zCsexOzIwmzlNth2S9gz0dGQV7/B7MTuY0UpagRe7mDl9YeYObGY1zdNpo3JvaSy3+szG6DJffB/pVw0wtwzWNmJ3IrKUpRrQtFNh5duJ2vDmeRPCiePw7vTMDFE2YL67CXwodJcGgd3PoyXPWQ2YncTopSXNKh9HwemLOVsxdKeHVCD+5IaGF2JGGmsmL44G5jCofhr0P/B8xOVCekKEWVvjyUycPztxES6M/CqQPp2yrK7EjCTLYiWDgRjm0wLiTvm2R2ojojRSkqNf/bEzy7Yi8dGkUwc0qC3GljdaUFsOBOOPkNjH0bek0yO1GdkqIUPzNr4zGeX7WPGzrG8q+7+hAeLP9MLK0kF+bfDmlbYfy70H2C2YnqnLwDxE/8d+85Xli9j5u7NOLtyX3kpI3VFZ+HuePh3C64/X3oMsbsRKaQohQ/2JV2gV8t2k6PZvV5c2JvKUmrK8yGuWMh8wDcMRc6DTc7kWmkKAUA+SVlPLJgO9FhwbyX1I96Qd4/4ouohYJMmDMGslNh4kJof5PZiUzl1k0GpdRQpdRBpVSqUurJSl6vr5RapZTaqZTaq5RKdmceUbXnVuwl7XwR/5zYi9gIGWzX0vLPwawRkHPUmCnR4iUJbixKpZQ/MA0YBnQBJimluly02MPAPq11T2Aw8DellHeOFe/Flm8/zdLtp/nVkPYkxHv2RPTCzXJPw/vDITfNmHO7zWCzE3kEd25R9gdStdZHtdY2YBFw8ZFgDUQoYwq+cCAHsLsxk7jIyewinlm+h4RWUTxyQzuz4wgzXTgJs4ZDQQbcsxTirzE7kcdw5zHKZsCpCo/TgAEXLfMWsBI4A0QAd2qtHYg6UWov59FF21EK3pjYS07eWFnOMZg9CkrzIHEFNO9rdiKP4s53RmVjLemLHt8K7ACaAr2At5RSkT/7RkpNVUptVUptlbm7XefF1fvZeeoCr03oIWNJWllWqrG7bSuAxJVSkpVwZ1GmARVvDG6OseVYUTKwVBtSgWNAp4u/kdZ6htY6QWudEBsb67bAVrJsexpzN59g6nVtGNpNhkqzrIwDxu52uQ2SVkPTXmYn8kju3PXeArRXSrUGTgMTgbsuWuYkMAT4SinVCOgIHHVjJks7mlnA5wcy2HI8h/UHM+nfuiF/uLWj2bGEWc7tMS4B8vOHKWsg7mfbKMLJbUWptbYrpR4BPgH8gZla671KqQedr78D/B8wSym1G2NX/QmtdZa7MlnZ/rN5jJu+kZIyB62iQxnbqymP39pJjkta1ZkdxsXkAfUgaRXEyIm8S3HrBeda67XA2ouee6fC52eAW9yZQUBucRkPzkshMiSQ/z52NS2j5XikpaWlwLxxEBwJSSuhYRuzE3k82ZzwcVprfr94J6fPFzN9ch8pSas7+a2xu10vCpLXSknWkBSlj3vny6P8b186Tw3vLBeTW93xr2HuOAiPgylroUFLsxN5DSlKH/b14Sxe++QAI3o04d5B8WbHEWY6sh7mTYD6zY0tyfrNzE7kVaQofVRqRj6/nJ9Cu7hw/npbD5QPTiEqaujwp8aguw3bGGe3IxqbncjrSFH6oOyCUu6dtZXgAD/+k9RPBt61soPrYNEkiO0IU1ZDuFyHfCWkKH1MWbmDX8xNIT2vhHcTE2jRUE7eWNa+FcZEYI26GWe3Q+UY9ZWSovQxy7adZuuJ87xyW3d6t5TJwCxr90ewOBma9YXE5cZZbnHFpCh9SFm5g7fWp9K9WX3G9pKD9Za1YwEsfQBaDoS7l0BIfbMTeT0pSh+ybPtpTuYU8esh7eXkjVWlzIblD0H8tTB5MQRHmJ3IJ0hR+gh7uYNp61Pp1iySIZ3jzI4jzPDdu7DqV9BuCNz1AQSFmZ3IZ0hR+ojFKWmcyC7i10M6yNakFW2aDmt/Dx2GwcQFECjzsLuSFKUPyC0u4/VPDtIvPoqbZGvSer7+B3zyFHQeDXfMgQCZ88jV5AI7H/DPTw+TU2Rj9qj+sjVpNV++Cuv/At1ug3EzwF/e0u4gv1Uv98XBDN7/5hiT+rekWzM5u2kZWhsFueE16DkJxkwzxpUUbiFF6cV2pV3g0QXb6dgogj+NuHiCS+GztIb/PQvfvAl9EmHkP8FPjqK5kxSlF9Jas3TbaZ5ftZcGYYHMnNKPekGyNWEJWsPHT8G3b0O/+2HYa1KSdUCK0stcKLLx60U7+PJQJgmtonhjYi+aNpAznJbgcMDa38HWmTDwIbj1JZBj0nVCitKLnMopImnmd6SdL+aF0V25Z2Ar/PzkjWIJjnLjGsnt82DQY3DT81KSdUiK0os8u2IPmQWlzH9gAP1kEF7rKLfDiodh1yK4/gkY/JSUZB2Tgxte4sC5PNYfzOSBa9tISVpJeZlx3/auRXDDM3DD01KSJpAtSi/x7y+PEhrkT+JVrcyOIuqK3QYfJcOB1XDzn2HQr81OZFlSlF5gy/EcVu48Q9JV8TQIDTI7jqgLZSWwOAkOfQxDX4GBvzQ7kaVJUXq4UzlFPDx/Gy2i6vHYze3NjiPqQlkxLJoMRz6DEX+HfveZncjypCg9WGZ+Kff851tKysqZe98AIkMCzY4k3M1WCAsnwrGvYPRb0OcesxMJpCg9Vm5xGYkzvyM9r5R59w+gY2MZV9DnlebD/Dvg1GYY92/oeafZiYSTFKUH0lozdc5WUjPy+U9SP/q2kmH8fV5JrjGd7OkUuO09Y5AL4TGkKD3QwfR8vj2Ww59GduG6DjJrns8rPg9zx8G5PXD7LOgy2uxE4iJSlB5o/YFMAEb2aGJyEuF2hdkwdwxkHoQ750HHoWYnEpWQovRA6w9m0KVJJI0iQ8yOItypIAPmjIGcozBpIbS7yexEogpyZ46HyS0qI+XEeW7sJCOV+7S8szBrBOQcM+a3kZL0aLJF6WGWbU+j3KG5QYrSd+WmwexRxhbl3UsgfpDZiUQ1pCg9xMnsIv68ei+f7s+ge7P69GrRwOxIwh3OnzBKsvg83LMMWvQ3O5GoASlKDzD7m+P8Ze1+AvwUTw3rRPKg1vjL8Gm+J+cozB4NpXmQuBya9TU7kaghKUqTfbDlJM+t3MuQTnH8ZVx3GteXEzg+KeuwsSVpL4WkVdCkp9mJxGWQojTRt0ezeWrpbq5tH8Pbd/clKEDOrfmkjP3GliQapqyGRl3NTiQuk7wzTbR022nCgwN4R0rSd53bbZzdVn4wZY2UpJeSd6dJTmQX8vnBDAa2iSYsWDbsfdKZ7TBrJASEQPJaiO1odiJxheQdaoINhzJ5aP42/BQ8cF0bs+MId0jbCnPHQ0h9mLIKouLNTiRqQYrSBNO/SKVBaCCLpg6keVSo2XGEq53YBPNvh7Bo48RNg5ZmJxK1JLveJkjNKGBQ2xgpSV907CuYdxtENILkdVKSPsKtRamUGqqUOqiUSlVKPVnFMoOVUjuUUnuVUl+6M48nyCm0kVVgo32jcLOjCFc78rmxJdmgBUxZC5FNzU4kXMRtu95KKX9gGnAzkAZsUUqt1Frvq7BMA2A6MFRrfVIp5fP37R1OzwegfSMZiNenHPovfHA3xLSHxBUQFmN2IuFC7tyi7A+kaq2Paq1twCJgzEXL3AUs1VqfBNBaZ7gxj+mKbeXM3HgMgI5SlL7jwBpYdBfEdTKOSUpJ+hx3FmUz4FSFx2nO5yrqAEQppb5QSqUopRLdmMdUJ7OLGDd9I//dl86TwzrJHTi+Yu8y+DARmvSAxJUQKnOu+yJ3nvWu7GZlXcnP7wsMAeoBm5RSm7XWh37yjZSaCkwFaNnS+w6Obz6azS/mpqC15v0p/Rjc0eePMFjDrsWwbCo07w+TF0NIpNmJhJu4c4syDWhR4XFz4Ewly3ystS7UWmcBG4Cf3QSrtZ6htU7QWifExnrX1AhFNjuPLdpBdFgQqx69RkrSV+xYAEsfgFaDjKHSpCR9mjuLcgvQXinVWikVBEwEVl60zArgWqVUgFIqFBgA7Hdjpjr39hdHOJdXwqsTetAqOszsOMIVUmbB8oegzfVw14cQLFcw+Dq37Xprre1KqUeATwB/YKbWeq9S6kHn6+9orfcrpT4GdgEO4D2t9R53Zaprp3KK+PeGo4zp1ZSEeDl25RO+exfW/h7a3WzMcRMox5qtwK135mit1wJrL3runYsevwa85s4cZnll3QH8leLJYZ3MjiJc4Zu34L9/hI4j4Pb3ISDY7ESijsidOW6SU2jj473nSLy6FU3q1zM7jqitr/5ulGSXMXDHbClJi5F7vd3k033plDs0o3rI3RleTWv48lX44iXofjuMfQf85W1jNfI37ibr9pyleVQ9ujaVs6FeS2v4/P/gq79Bz7tgzFvg5292KmEC2fV2Ma01S7elsTE1m6FdG6OUzH3jlbSG/z5jlGSfJBgzTUrSwqQoXWze5hP89sOdtIsLZ8qgeLPjiCuhNax7Aja9Bf0egJFvgJ+8VaxMdr1dyGZ3MP2LIyS0iuLDX1yFn8yk6H0cDljzW0h5H656BG55EWSvwPLkv0kXmrPpOGdzS3h0SHspSW/kKIeVjxolec1vpCTFD2SL0gW01kz/4givfXKQa9vHcF17GT3G65TbYfkvYfeHcP2TMPhJKUnxAylKF3h53QFmOO/AeXVCDzmB423Ky2DJ/bBvOdz4J7ju92YnEh5GirKWbHYHczYdZ1TPprxxZy8pSW9jL4WP7oUDq41d7asfNTuR8EBSlLX0zZEsSsocjOzRRErS25SVGGNJHv4Ehr0KA35hdiLhoaQoayGn0MbTS3fTPKoe17ST45JexVYEH0w25rkZ+Q9IuNfsRMKDXfZZb6WUv1JqsjvCeJs/rdhDVqGNtyf3JSxY/s/xGrZCWHAHHFlvXEguJSmqUWVRKqUilVJPKaXeUkrdogyPAkeBO+ououfacfICI7o3oXvz+mZHETVVkmdMJ3tiI4z7N/S+2+xEwgtcajNoLnAe2ATcDzwOBAFjtNY73B/Ns2mtycwvpVGkjEfoNYovwPwJcHob3PYf6Dbe7ETCS1yqKNtorbsDKKXeA7KAllrr/DpJ5uEuFJVhK3cQGyHDbXmFohyYOw7S9xrDpHUeZXYi4UUuVZRl33+itS5XSh2TkvxRRn4pAHFSlJ6vMAvmjIWsg8ao5B2Hmp1IeJlLFWVPpVQeP86mWK/CY621tvT4YceyCgApSo+Xnw5zxsD5YzBpEbQbYnYi4YWqLEqttYwpVYVD6fk8scS4LKhrMzmR47HyzsDs0ZB32pgErM31ZicSXqrKolRKhQAPAu0wJv+aqbW211UwT/bmZ4cBWPjAQMLlsiDPdOEUzB4FhZnGdLKtrjY7kfBil7qOcjaQAOwGhgN/q5NEHq6s3MGXhzK5pUsjWjQMNTuOqMz54zBrOBRlwz3LpSRFrV1qc6hLhbPe/wG+q5tInm3LsRzyS+zc1KWR2VFEZbKPGLvbtgJIXAHN+pidSPiAmp71tst9zIZP92cQFODHtTKUmufJPGTsbjvKIGkVNOlhdiLhIy5VlL2cZ7nBONNt+bPeWms+3Z/O1W2jCQ2SY5MeJX0fzBkNKEhaDY26mJ1I+JBLHaPcqbWOdH5EaK0DKnxuuZIESM0o4GROEUM6y263Rzm3G2aPBOUPU9ZISQqXu1RR6jpL4SWWbj+Nn4Jb5Pik5zi9DWaNhIAQSF4LsR3MTiR80KX2H+OUUr+t6kWt9d/dkMdj2ewOlqSkcUPHOLm/21Oc2gLzxkO9BsYxyah4sxMJH3WpovQHwvnxzhxL+2DrKTLyS0m8Ot7sKALgxDcw/3YIizVKskELsxMJH3apojyrtf5znSXxYCVl5bz1+WH6xUfJxGGe4NgGWHAnRDYzSjKyidmJhI+71DFK2ZJ0mrf5BOl5pfzulo4y3YPZUj8ztiQbtDKOSUpJijpwqaKU0QOAglI7b39xhGvbxzCwTbTZcazt0CewcCJEt4cpqyE8zuxEwiKqLEqtdU5dBvFUMzYcJbvQxu9v6Wh2FGvbvwoWTYa4LpC0EsLkEIioO5c9Z46VZOSX8N5XRxnRowk9WzQwO4517VkKHyZB017GbYmhDc1OJCxGivISpq8/gs3u4HHZmjTPzg9gyX3Qoj/cvdS4FEiIOiZFeQlbT+RwdbsY4mPCzI5iTdvnwbJfQKtBMPkjCLHkDWHCA0hRXkJGXimNI2UEc1NsnQkrHoY2g41Bd4PDzU4kLEyKsgrlDk1WQSlxEXIXTp379t+w+jfQ/lZj+oYgGfdTmEuKsgrZhaU4NMTJFmXd+uZfsO4P0GmkMRFYoPxHJcwnY4VVISPPmGUxNlyKss5seB0+/z/oMhZuew/8A81OJAQgRVml49mFADLdQ13QGr54Bb58BbrfAWPfBn/5pyk8h/xrrMLeM3kE+CnaN5KTCG6lNXz2Anz9D+g1GUb/C/xkAlDhWdx6jFIpNVQpdVAplaqUevISy/VTSpUrpSa4M8/l2HM6lw6NIggOkDet22gNn/zRKMm+yTD6LSlJ4ZHcVpRKKX9gGjAM6AJMUkr9bOhp53J/BT5xV5bLpbVm35k8ujWT6/bcxuGAtY/D5mnQ/xcw8h/gJ+cWhWdy57/M/kCq1vqo1toGLALGVLLco8ASIMONWS7LubwSsgttdG1a3+wovsnhgNWPwZZ34apHYNhfQUZlEh7MnUXZDDhV4XGa87kfKKWaAeOAd9yY47LtOW3MqSZblG7gKDcuJN82G679HdzyopSk8HjuLMrK/vVfPA/PG8ATWuvyS34jpaYqpbYqpbZmZma6Kl+V9p7JRSno1FiK0qXK7cYtiTsXwOCn4cY/SUkKr+DOs95pQMXx+ZsDZy5aJgFY5BwMNwYYrpSya62XV1xIaz0DmAGQkJDg9knP9pzOpXVMGGHBclGAy5SXGYNb7FsBQ56Da6ucjkkIj+POJtgCtFdKtQZOAxOBuyouoLVu/f3nSqlZwOqLS7KulTs0W46fZ2jXxmbG8C32UlicDAfXwC1/gasfMTuREJfFbUWptbYrpR7BOJvtD8zUWu9VSj3ofN2jjkt+b//ZPHKLy7i6nYxm7hJlJfDhPXD4vzDsNRgw1exEQlw2t+5baq3XAmsveq7SgtRaT3Fnlpr65kgWAFfJtA+1ZyuCRZPg6Jcw8g1ISDY7kRBXRA7CXeTr1GzaxoYRJ3N3105pgTG/zfGvYcw06D3Z7ERCXDG5wreCIpudzUezGdxRJq2qlZI8mHcbnNgI49+VkhReT7YoK9h0JBub3cENUpRXrviCUZJnd8CEmdB1nNmJhKg1KcoKPj+QQWiQP/1aR5kdxTsV5cDcsZC+D+6YA51GmJ1ICJeQonTSWvPFwUwGtYuRgTCuREGmUZJZh2HiAuhwi9mJhHAZOUbpdDijgNMXirmxk+x2X7b8czB7JGQfgbsWSUkKnyNblE6fHzDG5BjcMdbkJF4m7wzMHgV5Z2HyYmh9rdmJhHA5KUqn9Qcy6NQ4gib165kdxXtcOGWUZGEW3L0EWl1ldiIh3EJ2vQF7uYPtJy9wbfsYs6N4j5xj8P5w4wRO4nIpSeHTZIsSOHOhBFu5g3ZxMu1DjWQfMbYky4ogaQU07W12IiHcSooSOJpVAEDrGCnKamUehNmjwVEGSaugcXezEwnhdlKUwPEsY8bF+BiZcfGS0vfBnNGAgilrIK6z2YmEqBNyjBI4nl1EWJC/zOF9KWd3wqwR4BcAyWulJIWlSFECRzILaB0bhpLRtit3OsU4JhkYamxJxrQ3O5EQdcryRam1Zu+ZPLo0kWkfKnXyW5gzFkIaGFuS0W3NTiREnbN8UZ7NLSGn0Ea3ZjLj4s8c3wjzxkNYjFGSUa3MTiSEKSxflHvPGDMudm0qW5Q/cfQLmD8BIpvClLVQv7nZiYQwjeWLcuepC/j7KTrLrvePUj+FBXdCVLxxTDKyidmJhDCV5Yty64kcujaNJDRIrpQC4OA6WDjJOGGTtBrCZZAQISxdlDa7gx2nLtC3lYw/CcC+lfDB3dCoKySuhDCZN0gIsHhRpmYUUFLmoHdLKUr2LIHFU4zbERNXQGhDsxMJ4TEsXZSZBaUANK1v8YnEdi6CJfdDiwFwzzIIkSsAhKjI0kWZ7SzKaCvfkbNtLix7EOKvgbs/guAIsxMJ4XEsXpQ2AKLDg0xOYpIt78HKR6DtjXDXhxAUZnYiITySpYsyq7CUIH8/IoIteMZ789uw5nfQYagxx02gDFgsRFUsXZTZBTaiw4Osd4/3xn/Cx09Cp5Fwx1wItPgxWiGqYcFNqR9l5pdab7f7y9dg/YvQdTyMnwH+gWYnEsLjWboo0/NKaB5lkTEotYb1L8GGV6HHRBgzDfwt/dcvRI1Zetf7XF4Jjetb4Iy31vDpc0ZJ9r4bxk6XkhTiMlj23VJSVs6FojIaR/r48Tmt4ZOnYfN0SLgXhv8N/Cz9/6MQl82yRZmeVwJAI18uSocD1j1uXAY04EEY+gpY7cSVEC5g2aI8l2sUZWNfvSvH4YDVv4Ztc+DqX8HNf5aSFOIKWbconVuUPrnr7SiHFQ/DzoVw3eNwwx+lJIWoBcsWZZbzrpy4CB8rynI7LJtqDHJxwx/h+j+YnUgIr2fZoswtLkMpiAjxoV+B3QZL7oP9K+Gm5+Ga35idSAif4EMtcXnyisuICA7Az89HdkntpfBhEhxaB7e+BFc9bHYiIXyGpYsysp6P3JVSVmwMuJv6KQx/Hfo/YHYiIXyKZYvybG4JsRE+cLG5rdCYuuHYBhj1JvRNMjuRED7HslceH84ooH1cuNkxaqc0H+bfDse/Mu62kZIUwi0suUWZW1RGVkEp7by5KEtyjZJM2wrj34XuE8xOJITPcusWpVJqqFLqoFIqVSn1ZCWvT1ZK7XJ+fKOU6unOPN9LzcwH8N6iLD4Pc8bC6RS4/X0pSSHczG1blEopf2AacDOQBmxRSq3UWu+rsNgx4Hqt9Xml1DBgBjDAXZm+dzi9AID2cV447UFhNswdC5kHjLEkOw03O5EQPs+dW5T9gVSt9VGttQ1YBIypuIDW+hut9Xnnw81Aczfm+UFqRgEhgX40a+Blo3oXZMLsUZB50BiVXEpSiDrhzqJsBpyq8DjN+VxV7gPWuTHPD1IzC2gTE+5d11Dmn4NZIyDnKNz1AbS/2exEQliGO0/mVNZCutIFlboBoyivqeL1qcBUgJYtW9Y62OH0Avq28qK5vHNPG1uS+eeMmRLjK/01CSHcxJ1blGlAiwqPmwNnLl5IKdUDeA8Yo7XOruwbaa1naK0TtNYJsbGxtQqVX1LG6QvFdGzsJccnL5yEWcOhIAPuWSolKYQJ3FmUW4D2SqnWSqkgYCKwsuICSqmWwFLgHq31ITdm+cGhdOOMd+cmXlCUOUfh/eFQdB4SV0DLgWYnEsKS3LbrrbW2K6UeAT4B/IGZWuu9SqkHna+/AzwLRAPTnTMh2rXWCe7KBLD/rFGUHRtHuvPH1F5WqrG7bS+GpJXQtJfZiYSwLLdecK61Xgusvei5dyp8fj9wvzszXOzAuTwiQgJo6skD9mYcgDmjjXElk1ZD425mJxLC0ix3C+OBs/l0bhzpuXN5n9tjnN0GmLJGSlIID2CpotRac/BcvueeyDmzA2aPBP8gmLIW4jqZnUgIgcWK8vSFYvJL7XTyxBM5aSnG7nZQOCSvgZh2ZicSQjhZqigPOE/kdPK0EzknN8OcMRDSAJLXQsM2ZicSQlRgqaLcfzYPpfCsXe/jX8Pc8RAeB8nroEHtL6gXQriWpYpyz5lcWkeHER7sIaPLHVkP8yZA/ebGlmT9S93hKYQwi7WK8nQeXZvVNzuG4fD/YMGdxm72lDUQ0djsREKIKlimKC8U2Th9oZiuTT3g+OSBtbDoLojtAEmrILx2t2UKIdzLMkWZmV8KYP7QavtWwIf3QKNuRkmGRZubRwhRLcsUZXFZOQChQf7mhdj9ESxOhmZ9IXE51POiEYyEsDDrFKXNKMp6gSYV5Y4FsPQBY2CLu5dAiIccKxVCVMs6RencogwxY4syZTYsfwjir4XJiyHYgy5PEkJUyzJFWVJm0hbld+/Cql9BuyHGyORBYXX784UQtWaZoiw2oyg3TYe1v4cOw4w5bgK9bI4eIQRgoaLMK7YDEBpcR0X59T/gk6eg82i4Yw4EBNfNzxVCuJyH3KLifmnniwgO8CMmrA4K68tXYf1foNttMG4G+Fvm1yyET7LMO/hUTjHNo+q5d+ZFrY2C3PAa9JgIY6eDn4mXIwkhXMI6RXm+iBYNQ933A7SG/z0L37wJve+BUf+UkhTCR1jmGGVGfimNI900/YPW8PFTRkkm3Aej3pSSFMKHWGaLsqjU7p5RgxwOWPs72DoTBj4Et74EnjrNhBDiiliiKB0OTaGtnFBXF6Wj3LhGcvs8GPRruOkFKUkhfJAlivL7ayjDXXlpULkdVjwEuz6A6/4ANzwtJSmEj7JEURbanNdQBrlodcvLYOlU2LsUbngGrn/cNd9XCOGRLFGUuUVlAETWC6z9N7Pb4KNkOLAabv6zscsthPBplijKU+eLABeMRVlWAouT4NDHMPQVGPhLF6QTQng6SxTl2dwSoJZFWVYMiybDkc9gxN+g3/0uSieE8HSWKMrzhTYAosKucNfbVggLJ8Kxr2D0v6BPogvTCSE8nSWKMqewjPDgAIIDruCsd2k+zL8DTm2Gce9Az4muDyiE8GiWKMqsglIahgVd/heW5BrTyZ5OgdveMwa5EEJYjiWK8tT5oss/PlmUA/PGw7ndcPss6DLaLdmEEJ7PEvd6n8oppkXDyyjKwmyYMxrS98Kd86QkhbA4n9+iLLaVk1VQSouoGo4cVJABc8ZAzlGYuBDa3+TegEIIj+fzRXkmtxiAZlE12KLMO2tsSV44Zcxv02awe8MJIbyCzxdlRl4pAI2qG2ItNw1mj4L8dGM62fhBdZBOCOENfL4oMwuMooyNuMQUEOdPGCVZfB7uWQYtB9RROiGEN/D5orTZHcAlZl/MOQqzRoEtHxKXQ7O+dRdOCOEVfL4oC0uNkYOCAys5wZ912NiStJdC0ipo0rOO0wkhvIHPF+XZ3BIC/dXPZ1/M2A+zRwMapqyGRl1NySeE8Hw+fx3ludxiGkWG/HT2xXO7YdYIY6DdKWukJIUQl+TzRXkmt4Sm9StcGnRmO8waCf7BMGUtxHY0L5wQwiv4fFFm5pcSF+nc7U7bCrPHQHAkJK+FmHbmhhNCeAW3FqVSaqhS6qBSKlUp9WQlryul1JvO13cppfq4OoPd4SDI3w9ObII5YyE0CpLXQMPWrv5RQggf5baiVEr5A9OAYUAXYJJSqstFiw0D2js/pgJvuyNL28LtMO82iGgEyeugQUt3/BghhI9y5xZlfyBVa31Ua20DFgFjLlpmDDBHGzYDDZRSTVwZIqF8Bw+cegIatDCOSUY2deW3F0JYgDuLshlwqsLjNOdzl7sMSqmpSqmtSqmtmZmZNU9QVswfbW+SFdQMklYbW5RCCHGZ3HkdZWWTXOsrWAat9QxgBkBCQsLPXq9SYD0KbltEYFRzCI+t8ZcJIURF7izKNKBFhcfNgTNXsEytxHcb6MpvJ4SwIHfuem8B2iulWiulgoCJwMqLllkJJDrPfg8EcrXWZ92YSQghLpvbtii11nal1CPAJ4A/MFNrvVcp9aDz9XeAtcBwIBUoApLdlUcIIa6UW+/11lqvxSjDis+9U+FzDTzszgxCCFFbPn9njhBC1JYUpRBCVEOKUgghqiFFKYQQ1ZCiFEKIakhRCiFENaQohRCiGsq4lNF7KKUygROX+WUxQJYb4pjBV9bFV9YDZF081eWuSyutdaWDQnhdUV4JpdRWrXWC2TlcwVfWxVfWA2RdPJUr10V2vYUQohpSlEIIUQ2rFOUMswO4kK+si6+sB8i6eCqXrYsljlEKIURtWGWLUgghrphPFaUnTI/rCjVYj8nO/LuUUt8opXqakbMmqluXCsv1U0qVK6Um1GW+y1GTdVFKDVZK7VBK7VVKfVnXGWuiBv++6iulVimldjrXw2PHiVVKzVRKZSil9lTxumve81prn/jAGBz4CNAGCAJ2Al0uWmY4sA5jrp6BwLdm577C9bgaiHJ+PswT16Om61Jhuc8xxi6dYHbuWvy9NAD2AS2dj+PMzn2F6/E08Ffn57FADhBkdvYq1uc6oA+wp4rXXfKe96UtSo+YHtcFql0PrfU3WuvzzoebMeYa8kQ1+TsBeBRYAmTUZbjLVJN1uQtYqrU+CaC19sT1qcl6aCBCKaWAcIyitNdtzJrRWm/AyFcVl7znfakoXTY9rskuN+N9GP9jeqJq10Up1QwYB7yDZ6vJ30sHIEop9YVSKkUplVhn6WquJuvxFtAZY6K/3cCvtdaOuonnci55z7t1Kog65rLpcU1W44xKqRswivIatya6cjVZlzeAJ7TW5cYGjMeqyboEAH2BIUA9YJNSarPW+pC7w12GmqzHrcAO4EagLfA/pdRXWus8N2dzB5e8532pKD1ielwXqFFGpVQP4D1gmNY6u46yXa6arEsCsMhZkjHAcKWUXWu9vE4S1lxN/31laa0LgUKl1AagJ+BJRVmT9UgGXtHGQb5UpdQxoBPwXd1EdCnXvOfNPhjrwoO6AcBRoDU/HqTuetEyI/jpgd3vzM59hevREmPmyqvNzlvbdblo+Vl47smcmvy9dAY+cy4bCuwBupmd/QrW423geefnjYDTQIzZ2S+xTvFUfTLHJe95n9mi1D4yPW4N1+NZIBqY7twSs2sPHMighuviFWqyLlrr/Uqpj4FdgAN4T2td6WUrZqnh38n/AbOUUrsxCuYJrbVHjiiklFoIDAZilFJpwHNAILj2PS935gghRDV86ay3EEK4hRSlEEJUQ4pSCCGqIUUphBDVkKIUQohqSFEKr+UcbWhHhY945+g9uUqp7Uqp/Uqp55zLVnz+gFLqdbPzC+/hM9dRCksq1lr3qviEUioe+EprPVIpFQbsUEqtdr78/fP1gO1KqWVa6411G1l4I9miFD5LG7cSpmDcr1zx+WKMe5k9bUAU4aGkKIU3q1dht3vZxS8qpaIxblvbe9HzUUB7YEPdxBTeTna9hTf72a6307VKqe0YtxG+4rxFb7Dz+V1AR+fz5+osqfBqUpTCF32ltR5Z1fNKqQ7A185jlDvqOJvwQrLrLSxHG+NDvgw8YXYW4R2kKIVVvQNcp5RqbXYQ4flk9CAhhKiGbFEKIUQ1pCiFEKIaUpRCCFENKUohhKiGFKUQQlRDilIIIaohRSmEENWQohRCiGr8PzahvgJ7LKAdAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 360x360 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.figure(figsize=(5,5))\n",
"plt.plot(df_scores.fpr, df_scores.tpr, label=\"model\")\n",
"plt.plot([0, 1], [0,1], label=\"random\")\n",
"plt.xlabel(\"FPR\")\n",
"plt.ylabel(\"TPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "dc6adf70-1e73-436a-bb01-8b128341cf56",
"metadata": {},
"source": [
"* We can uses sklearn to plot the ROC curve"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "3608c17c-7052-48fc-844a-81eef9a46db3",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import roc_curve"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "f655cabe-ce25-41a1-848b-f9fdbae97540",
"metadata": {},
"outputs": [],
"source": [
"fpr, tpr, thresholds = roc_curve(y_val, y_pred)"
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "54c46165-8b39-4d68-9051-b26d736d9d77",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAE9CAYAAABtDit8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAojklEQVR4nO3dd5hV1b3G8e+PoSO9WCgCEQsEEBjBDkqigMIINorOMLHEqGk3MZpmSbzGG41JvIIEDdJFkd70JkbFBgKKoiCKgDgUqQIibZh1/9gDjuP0OfvsffZ+P88zzzPnnM2c357ystZea69lzjlERKR4VYIuQEQk7BSUIiKlUFCKiJRCQSkiUgoFpYhIKRSUIiKlqBp0AeXVpEkT17p166DLEJGIWbZs2XbnXNOiXku5oGzdujVLly4NugwRiRgz+7S419T1FhEphYJSRKQUCkoRkVKk3DXKohw+fJicnBwOHDgQdCmhVrNmTVq0aEG1atWCLkUkpUQiKHNycqhbty6tW7fGzIIuJ5Scc+zYsYOcnBzatGkTdDkiKSUSXe8DBw7QuHFjhWQJzIzGjRur1S1SAZEISkAhWQb6HolUjG9BaWZjzGyrmb1fzOtmZo+a2Roze8/MuvpVS6pp3bo127dvr/QxIpIYfrYoxwJ9Sni9L9Au/+Nm4HEfaxERqTDfBnOccwvNrHUJh2QA4523xPoiM2tgZic65zb7VZOf1q9fT58+fTj//PNZtGgRnTt3Jjs7m3vuuYetW7cyadIkTjnlFH7wgx+wdu1aateuzejRo+nUqRM7duxgyJAhbNu2je7du1Nw1fmJEyfy6KOPcujQIXr06MHIkSNJS0sL8ExFKm7y4g3MWr7R1/dIc7lctm8G6065nt9ldEnI1wxy1Ls58FmBxzn5z30rKM3sZrxWJ61atUpKcRWxZs0apk6dyujRoznrrLOYPHkyr732GrNnz+aBBx6gZcuWdOnShZkzZ/Kf//yHzMxMli9fzn333cf555/P3Xffzbx58xg9ejQAq1at4plnnuH111+nWrVq3HrrrUyaNInMzMyAz1SkeCWF4eJ1OwHo0aaRL+9d1R3iZ7v+xFkH32TKnnZA6gdlUSMLRW7g45wbDYwGSE9PL3GTn/vmfMDKTXsqX10B7U+qxz39O5R6XJs2bejYsSMAHTp0oHfv3pgZHTt2ZP369Xz66adMmzYNgIsvvpgdO3awe/duFi5cyPTp0wG47LLLaNiwIQAvvvgiy5Yt46yzzgJg//79NGvWLKHnJqkpGS2ziiopDHu0aUTGmc0Z2sOHBs/hA/Ds9bDlTej3MIO735SwLx1kUOYALQs8bgFsCqiWhKhRo8axz6tUqXLscZUqVcjNzaVq1W9/u4+ORBc1Iu2cIysriz/96U8+VSyponAw+t0yqwxfw7A4h76CKUNg7SvQ/+/QbXhCv3yQQTkbuN3MpgA9gN2JuD5ZlpZfUC688EImTZrE73//e15++WWaNGlCvXr1jj3/u9/9jgULFrBr1y4AevfuTUZGBj//+c9p1qwZO3fuZO/evZx88skBn4n4rbRgDCSMwurgl/D0YFj/GmSMgC7DEv4WvgWlmT0N9AKamFkOcA9QDcA5NwqYD/QD1gBfAdl+1RIW9957L9nZ2XTq1InatWszbtw4AO655x6GDBlC165d6dmz57HrsO3bt+f+++/nkksuIS8vj2rVqjFixAgFZQSU1nVWMJbRgT0w6WrIWQKDnoBOV/vyNpZq+3qnp6e7wutRrlq1ijPOOCOgilKLvlfBqEjXWcFYiv1fwMQrYfNyuPKf0OGKSn05M1vmnEsv6rVI3OstEnazlm9k5eY9tD+xHqAWYqV9tRMmXAGfr4RrxsPpl/n6dgpKER8dbUkeDclnfnhO0CWlvi+3eSG5/WMYPBlOvcT3t1RQivhk8uIN/GbGCuDrFqRU0t7PYfwA2PUpDJ0C37k4KW+roBQpo/LOXTx6HfKBgR3VxU6EPZtgXH/YsxmGTYU2FyTtrRWUIkUoKhTLO3dR1yET6IvPvJDctx2unw6tzk7q2ysoRQop3GU+SsEXkJ3rYNwAOLAbMmdCiyIHpn2loAypo9vyNmnSJOhSIqUs3Wd1mUNkxydeS/LwV5A1C05KzL3b5aWg9IFzDuccVapEZl3klFNcIJal+6yWY0hsW+21JPMOQ9YcOKFjYKUoKBNk/fr19O3bl4suuog333yTM888kxUrVrB//36uuuoq7rvvPsBrKWZlZTFnzhwOHz7M1KlTOf3000tcau2RRx5hzJgxANx444387Gc/K9Oybt27dw/ke5FM5Q1EhWCK+HylN7qNwfB50CzYmyQUlAm0evVqnnrqKUaOHMnOnTtp1KgRR44coXfv3rz33nt06tQJgCZNmvD2228zcuRIHn74YZ588slil1pbtmwZTz31FIsXL8Y5R48ePejZsycNGzYsdVm3mTNnBvjd8NfRgFQgRtDmd2H8FVC1hteSbNIu6IoiGJQL7oItKxL7NU/oCH0fLPWwk08+mbPP9kbjnn32WUaPHk1ubi6bN29m5cqVx4Jy0KBBAHTr1u3Y8mrFLbX22muvMXDgQOrUqXPs37766qsMGDCg1GXdouzoJG4FYsRsXAYTBkL1upA1Gxp/J+iKgCgGZYCOhtm6det4+OGHWbJkCQ0bNmT48OHf2P3w6PJraWlp5ObmHnu+uKXWilPasm5RUriLrTtdIuizt7x7t2s19FqSDcOz+Ev0grIMLT+/7dmzhzp16lC/fn0+//xzFixYQK9evUr8N8UttXbhhRcyfPhw7rrrLpxzzJgxgwkTJiThLMKjqOk67U+spztdomT96zD5GjjueK8lWb9F0BV9Q/SCMgQ6d+5Mly5d6NChA23btuW8884r9d8Ut9Ra165dGT58+LGBmRtvvJEuXbpEvmt9VMGQ1HSdiFr7Mjw9xAvHrDlQ94SgK/oWLbMWM6n0vVJIxsCaf8OUYdCoLWTOguOC2+pEy6xJyih4LVITvyNu9QJ4NhOangbXz4I6jYOuqFgKSgmNwtciNaIdYStnw3PZ3oyS66ZD7fDt/VOQglJC42hLUi3IiHt/Gky7CZp3g+ueg5r1g66oVJEJSudckdNr5GthuB5d0r3WR+dFKiQj7N0pMPNH0PJsGPYs1KgbdEVlEomgrFmzJjt27KBx48YKy2I459ixYwc1a9YM5P1Lu5MGNOUn8t6eALN/7K0jOWQKVK8TdEVlFomgbNGiBTk5OWzbti3oUkKtZs2atGiRvPlpRQ3M6LpjTC35J8z7L/hObxg8CarVCrqicolEUFarVo02bdoEXYYUoIEZOWbRKHj+Tji1D1w9DqoF06upjEgEpYSL5j/KMa//Hf51N5zRH64cA1WrB11RhSgoJaEUknLMKw/BS/dDh0EwaDSkVQu6ogpTUErCKCQFAOfgpQdg4Z+h02DIGAFpqR01qV29hIZCUgAvJP99L7z+N+hyHfR/FKqkBV1VpSkoJSE0WVxwDl74DSwaCek3QL+HISLboSgopcIKTv/RZPGYy8uDBXfAkiehx4+gz58gQnOaFZRSbkVNHtdk8RjLy4O5P4W3x8O5P4Hv/yFSIQkKSimnwvMjNTcy5vKOwKzb4N2n4cI74KLfRi4kQUEp5aABG/mGI7kw44fw/nNeQPb8VdAV+UZBKWWmARs5JvcQTLsBVs2G790H5/8s6Ip8paCUUh29JqkBGwEg9yA8mwUfLYBL/wTn3Bp0Rb5TUEqJiromKTF2eD88c523hUO/h6H7TUFXlBQKSilS4ZFtdbeFQ1/B04Nh3UJvInm3rKArShoFpXyLRrblWw5+CZOvhQ1vwBWPw5lDgq4oqRSU8i0atJFvOLAbJl0NOUth0BPQ8aqgK0o6BaUUSYM2AsD+XTBhEGx5D65+CtpnBF1RIBSUckzB0e32J9YLuhwJ2r4dMOEK2PYhXDMBTu8XdEWBUVAKoNFtKeTLbTA+A3asgcFPQ7vvBV1RoHwNSjPrA/wdSAOedM49WOj1+sBEoFV+LQ87557ysyYpmq5LyjF7t8C4AfDFBm+nxLa9gq4ocL6tgWRmacAIoC/QHhhiZu0LHXYbsNI51xnoBfzFzFJzrfgUNnnxBhav26nrkgK7N8JT/WB3jrfntkIS8DEoge7AGufcWufcIWAKUPhKsAPqmrfH7HHATiDXx5qkCEdbk+pux9wXG2BsP/hyK1w/HVqfH3RFoeFn17s58FmBxzlAj0LHPAbMBjYBdYFrnXN5PtYkhag1KQDsXAfj+sPBPZA5C1p0C7qiUPGzRVnUWkuu0ONLgeXAScCZwGNm9q3hVjO72cyWmtlS7d2dOAUHcNSajLHta7zu9qEvIXO2QrIIfgZlDtCywOMWeC3HgrKB6c6zBlgHnF74CznnRjvn0p1z6U2bNvWt4DjRkmkCwNYPve72kUOQNRdOOjPoikLJz673EqCdmbUBNgKDgaGFjtkA9AZeNbPjgdOAtT7WFGsFt27QPdzClve9KUBV0mD4PGj2rTaK5PMtKJ1zuWZ2O/AC3vSgMc65D8zslvzXRwF/BMaa2Qq8rvqdzrntftUUdwUnk+se7pjbtNybTF61FmTNgSanBF1RqPk6j9I5Nx+YX+i5UQU+3wRc4mcN8u07bp754TlBlyRBylkGEwdCjXqQNRsatQ26otDTnTkRpztu5Bs2LIaJV0Kdxl5LsoF6FGWhoIwwDdjIN6x/DSZdA3VP8EKyvv7TLKto7E4uRdJtiXLMJy/BxKugfgvInq+QLCcFZURpIrkc8/G/vUV3G7X1RrfrnhB0RSlHQRlBmkgux6xeAFOGQNPTYPhcOE7zkCtCQRkxui4px6yc5W0Edvx3vdHt2o2CrihlKSgjRtclBYAVz8HUbGjeDTJnQq2GQVeU0hSUEaLrkgLA8skw/SZodTZcNw1q1g+6opSnoIwQLZcmLBsHM2+F1hfAsKlQo27QFUWCgjIi1JoU3noC5vwETukNQ5+B6nWCrigyFJQRodZkzL05Eub/Ek7tC4MnQ7VaQVcUKQrKCFFrMqZe+yu88Gs4YwBcMx6q1gi6oshRUEbA0W63xNArf4Z/3wvfvRKuegqqasspP+he7xR2dFWgoyGpbneMOAcv/TcsfAg6D4GMEd66kuILBWUKO7p0mtaWjBnn4F93wxuPQtdMuPzvUEWdQz8pKFOQ1peMMefg+V/D4sfhrBuh70MKySRQUKYYrS8ZY3l5MP8XsHQMnH0rXPoAWFF7+EmiKShTiO7jjrG8I94cyXcmwnk/g+/dq5BMIgVlCtF93DF1JBdm3QbvTYGed0KvXyskk0xBmSJ0501MHTkM02+GD6bDRb+DnncEXVEsKShThO68iaHcQ/BcNnw4F77/Bzjvp0FXFFsKyhSi1mSMHD4AU7Pgo+ehz4Nw9o+CrijWFJQhV3gqkMTA4f0wZRh88iJc9gicdUPQFcWegjLkCoakut0xcGgfPD0Y1r0KAx6DrtcHXZGgoAy1ggM4mlQeAwf3etvJfrYIBv4DOl8bdEWST0EZUtogLGYO7Pa2k924DK580lvkQkJDQRlCmlgeM/t3wYSBsOV9uHostB8QdEVSiIIyhDSxPEb27YAJGbBtNVw7EU7rE3RFUgQFZchoYnmMfLkVxmfAzrUw5Gk45XtBVyTFUFCGjCaWx8SezTB+AHzxmbe/TdteQVckJVBQhohakzGxOwfG9fdalNdNg9bnBV2RlEJBGRIa5Y6JXZ96Ibl/F1w/A1p2D7oiKQMFZQholDsmdq6FcQPg4B7InAnNuwVdkZSRgjJgCsmY2P6x15LMPQhZc+DEzkFXJOWgoAyQQjImtq7yWpI4GD4Xju8QdEVSTtpsI0CaLxkDW1bA2MvAqsDweQrJFKWgDIhGuGNg0zsw9nKoWhOy50PT04KuSCpIQRkAjXDHQM5SGJcBNep5Idn4O0FXJJWgoAyAutwR9+mbMP4KqN0QsudBw9ZBVySVpKAMiLrcEbXuVZh4JdQ9HrIXQAP9jKPA16A0sz5mttrM1pjZXcUc08vMlpvZB2b2ip/1iPjqk//ApKuhQUsYPh/qnRR0RZIgvk0PMrM0YATwfSAHWGJms51zKwsc0wAYCfRxzm0ws2Z+1SPiq4/+D565Dpq0g8xZUKdJ0BVJAvnZouwOrHHOrXXOHQKmABmFjhkKTHfObQBwzm31sZ7ATV68gWv/8SYrN+8JuhRJpA/nwZSh0Ox0bzK5QjJy/AzK5sBnBR7n5D9X0KlAQzN72cyWmVmmj/UE6uhI9+J1O7X/TZR8MAOezYQTO0HmbKjdKOiKxAd+3pljRTzninj/bkBvoBbwppktcs599I0vZHYzcDNAq1apd3Fcd+BE1HtTYcbN0KI7DJsKNbVLZlT52aLMAVoWeNwC2FTEMc875/Y557YDC4Fv3QTrnBvtnEt3zqU3bdrUt4L9oulAEbR8Mky/CU4+z1sqTSEZaX4G5RKgnZm1MbPqwGBgdqFjZgEXmFlVM6sN9ABW+VhTUhW8JqnpQBGybCzMvBXa9oShz0KN44KuSHzmW9fbOZdrZrcDLwBpwBjn3Admdkv+66Occ6vM7HngPSAPeNI5975fNSWb9uSOoLeegPm/hFO+7+1xU61m0BVJEvi6epBzbj4wv9Bzowo9fgh4yM86gtT+xHrakzsq3ngM/u+3cNplcPVTULVG0BVJkujOHJ8cXfRCIuLVR7yQbJ8B14xTSMaM1qP0iTYJiwjn4JU/w8sPQMer4YpRkKY/m7jRT9xHGsBJcc7Bf/4Ir/4FOg+FjMegSlrQVUkA1PX2gbrdEeAc/N/vvJDsmgUZIxSSMaYWZYJprckIcA4W3Alv/QPOugn6/hmqqE0RZwrKBNIdOBGQlwfz/guWPQXn3A6X3A9W1E1mEicKygRRSEZA3hGY/RNYPhHO/zn0vkchKYCCMmF0m2KKO5ILM38EK56FnndBr7sUknKMgjIBtFFYijtyGKbdCCtnwsW/hwt/GXRFEjIKykrS4E2Kyz0Iz/0APpzrXY8898dBVyQhpKCsJHW5U9jhA95akh+/4I1s9/hh0BVJSCkoK0Fd7hR26Ct4Zpi3z83lf4X0HwRdkYRYuSeHmVmamQ3zo5hUo9sUU9ShfTD5GvjkJW8iuUJSSlFsUJpZPTP7tZk9ZmaXmOfHwFrgmuSVGG5qTaaYA3u87WQ/fR0G/gO6XBd0RZICSup6TwB2AW8CNwJ3ANWBDOfccv9LE0mw/V/ApKtg49tw5T/hu4OCrkhSRElB2dY51xHAzJ4EtgOtnHN7k1KZSCJ9tRMmDITPP/CWSTujf9AVSQop6Rrl4aOfOOeOAOsUkl/TwhcpZN92GDcAtq70ViVXSEo5ldSi7Gxme/h6N8VaBR4751ysd1PSQE6K2Ps5jM+AXetgyBQ4pXfQFUkKKjYonXNaU6oIkxdvOLYXjgZyQm7PJq8luWejtwlY255BVyQpqtigNLOawC3AKXibf41xzuUmq7Cw0oZhKeKLz2Bcf9i3zdtO9uRzg65IUlhJXe9xeNcpXwX6AR2AnyajqLDThmEht2u9F5L7v4DrZ0LLswIuSFJdSUHZvsCo9z+Bt5JTkkgl7PjE624f+hIyZ0HzrkFXJBFQUlAWHPXONS05JWG37SOvJZl3GLLmwImdgq5IIqKkoDwzf5QbvJFujXpLeH2+EsYPAAyy5sLx7YOuSCKkpHmU7zrn6uV/1HXOVS3weSxDUnMnQ2rLChh3OVgaDJ+nkJSEKykoXdKqSBGaOxlCG9+GsZdD1ZqQPR+anhp0RRJBJXW9m5nZfxX3onPuER/qCS0tqRZCny2BiYOgVgPvmmTD1kFXJBFVUlCmAcfx9Z05saVVzEPo0zdg0tVQp6kXkg1aBl2RRFhJQbnZOfeHpFUSYlrFPGTWLYTJ10K95l5I1jsx6Iok4kq6Rhn7lmRB6nKHxJoXvZZkg5O9a5IKSUmCklqUsV89oOB93e1PjOVAf7h89AI8cx00OQ0yZ0KdJkFXJDFRbIvSORf7eTC6rztEVs2BKcOgWXvImq2QlKTS5mKl0H3dIfD+dG/f7eZdYdhz3ii3SBKVe3MxkaR69xmYdgO07A7XTVdISiAUlBJe70yEGT+Ek8/zWpI1dZ1YgqGglHBaOgZm3QZte3mL7tY4LuiKJMYUlBI+i/8Bc38O7S71tm+oXjvoiiTmFJQSLm/8Lyz4FZx+ubcRWLWaQVckolFvCZGFD8N//gjtr4Arn4S0akFXJAIoKCUMnIOXH4RXHoSO18AVj0OafjUlPPTbKMFyDl68D177K5w5DAb8L1TRBqASLr5eozSzPma22szWmNldJRx3lpkdMbOr/KynPLRIbxI4By/81gvJbtkw4DGFpISSb0FpZmnACKAv0B4YYmbfWno6/7j/AV7wq5aK0CK9PsvLg/l3wKIR0P2HcPlfoYrGFiWc/PzN7A6scc6tdc4dAqYAGUUc92NgGrDVx1oqRCsG+SQvD+b+DJY8AefcDn3/B7R5nYSYn0HZHPiswOOc/OeOMbPmwEBglI91SJjkHfEmkr89Di74BVxyv0JSQs/PoCzqt7/wPjx/A+50zh0p8QuZ3WxmS81s6bZt2xJVX7F0fdInR3K9WxLfnQy9fgMX/14hKSnBz1HvHKDg+vwtgE2FjkkHpuTvGd4E6Gdmuc65mQUPcs6NBkYDpKen+77pma5P+uDIYW9xi5WzoPc9cEGx2zGJhI6fQbkEaGdmbYCNwGBgaMEDnHNtjn5uZmOBuYVDMii6PplAuQdhajasngeX/Dece3vQFYmUi29db+dcLnA73mj2KuBZ59wHZnaLmd3i1/tWlrrdCXb4gLcq+ep50PchhaSkJF8nnDvn5gPzCz1X5MCNc264n7WUlbrdCXToK5gyBNa+Apf/DdKzg65IpEJ0Z04R1O1OgINfwtODYf1rkDECugwLuiKRClNQSuId2OPtlJjzFgx6AjpdHXRFIpWiWyEK0PXJBNj/BUwYCBuXwlVjFJISCWpRFqDrk5X01U6YcAV8vhKuGQ+nXxZ0RSIJoaAsRNcnK+jLbV5Ibv8YBk+GUy8JuiKRhFHXO5+63ZWwdwuMuxx2fAJDpygkJXLUosynbncF7dkE4/rDns0wbCq0uSDoikQSTkFZgLrd5fTFZ15I7tsO102Dk88JuiIRXygopWJ2roNxA+DAbsicCS3Sg65IxDe6RomuT5bbjk9g7GVwaC9kzVJISuSpRYmuT5bLttVeSzLvMGTNgRM6Bl2RiO8UlPl0fbIMPl8J4wcABsPnQbMzgq5IJCnU9Zay2fyu192uUhWy5yskJVYUlFK6jcu80e1qtb2WZJN2QVckklTqekvJNiyGSVdBrYbeNcmGJwddkUjSqUUpxVv/OkwcBHWaeN1thaTEVOyDUlODirH2Za8lWe8kGD4f6rcIuiKRwMQ+KDU1qAhr/g2Tr4WGrb1rkvVODLoikUDFPihBU4O+YfUCeHqIN2CTNReOaxZ0RSKBU1DK11bO9jYCO74DZM6GOo2DrkgkFBSU4nl/GkwdDid1gcxZULtR0BWJhEasg1IDOfnenQLTboSWPeD6GVCzftAViYRKrINSAznA2xNgxi3Q+ny47jmoUTfoikRCJ9ZBCTEfyFnyJMy+Hb5zMQx9FqrXCboikVCKfVDG1qLHYd4v4NQ+3h431WoFXZFIaCko4+j1v8Pzd8Hpl8M1E6BazaArEgk13esdN688BC/dDx0GwaDRkFYt6IpEQk9BGRfOwUsPwMI/Q6fBkDEC0vTjFykL/aXEgXPw73u8LneX66D/o1AlLeiqRFKGgjLqnIMXfgOLRkL6D6DfX6CKLk2LlIeCMsry8mDBHd40oB63QJ8HwSzoqkRSTmybFpG/KycvD+b+1AvJc3+ikBSphNi2KCN9V07eEZh1G7z7NFx4B1z0W4WkSCXENighonflHMmFGTd7i1xc9Fvo+augKxJJebEOysjJPQTTboBVs+F798L5Pw+6IpFIUFBGRe5BeDYLPloAlz4A59wWdEUikaGgjILD+70Fd9f8G/o9DN1vCroikUhRUKa6Q/u8rRvWLfQmknfLCroikchRUKayg3u9TcA2vAlXjIQzhwZdkUgkKShT1YHdMOlqyFkKg56AjlcFXZFIZPk64dzM+pjZajNbY2Z3FfH6MDN7L//jDTPr7Gc9kbF/F4y/AjYug6ufUkiK+My3oDSzNGAE0BdoDwwxs/aFDlsH9HTOdQL+CIz2q56CUvqunH07YNwA+Px9by3J9hlBVyQSeX62KLsDa5xza51zh4ApwDf+qp1zbzjnduU/XAS08LGeY1L2rpwvt8G4/rBttbcq+en9gq5IJBb8DMrmwGcFHufkP1ecG4AFPtbzDSl3V87eLTD2Mti5FoY+A+2+H3RFIrHh52BOUTcXuyIPNLsILyjPL+b1m4GbAVq1SqFwS5TdG72W5N4t3k6JrYv8NomIT/xsUeYALQs8bgFsKnyQmXUCngQynHM7ivpCzrnRzrl051x606ZNfSk2tL7YAGP7wZdb4frpCkmRAPgZlEuAdmbWxsyqA4OB2QUPMLNWwHTgeufcRz7Wkpp2roWn+sFXuyBzFrQ6O+iKRGLJt663cy7XzG4HXgDSgDHOuQ/M7Jb810cBdwONgZHmLQOW65xL96umlLJ9jdfdzt0PWbPhpDODrkgktnydcO6cmw/ML/TcqAKf3wjc6GcNhR2dGtSjTaNkvm35bP0Qxg/w1pXMmgsnfDfoikRiLXYrnId+atCW973RbYDh8xSSIiEQu6CEEE8N2rQcxl0OadVh+HxodnrQFYkIMQvKUN+Rk7PM625XPw6y50GTU4KuSETyxSooQ9vt3rAIxmdAzQaQPR8atQ26IhEpIFZBCSHsdq9/DSYMguOaQfYCaBCi2kQEiGFQhsonL8HEq6B+C68lWT9kLV0RARSUwfn4X96iu43aeqPbdU8IuiIRKYaCMggfzocpQ6HpqZA1B46L2W2ZIilGQZlsK2fBs9fD8d/1QrJO46ArEpFSKCiTacVzMDUbmneDzJlQq2HQFYlIGSgok2X5ZJh+k7ewxXXToGb9oCsSkTKKTVAGOtl82TiYeSu0vgCGTYUadYOpQ0QqJDZBGdhk87eegDk/gVN6eyuTV6+T3PcXkUqLTVBCAJPN3xwJ838Jp/b19ripVit57y0iCROroEyq1/4KL/wazhgA14yHqjWCrkhEKsjX9Shj65U/w0v/Dd+9EgaOhjR9m0VSmf6CE8k5LyAXPgSdBsMVI6FKWtBViUglKSgTxTn4193wxqPQ5Xro/3eFpEhEKCgTwTl4/tew+HFIvwH6PQxVdPlXJCoUlJWVlwfzfwFLx8DZt8KlD4AVtaW5iKQqBWVl5B3x5ki+MxHO+yl87z6FpEgEKSgr6kguzLoV3nsGLvwVXPQbhaRIRCkoK+LIYZh+M3wwHS76HfS8I+iKRMRHCsryyj0Ez2XDh3Ph+3/wutwiEmmxGJpN2IIYhw94a0l+OBf6PKiQFImJWLQoE7IgxuH9MGUYfPIiXPYXOOvGBFUnImEXi6CESi6IcWgfPD0Y1r0KA/4XumYmtjgRCbXYBGWFHdwLk66BzxbBwFHQeXDQFYlIkikoS3Jgt7ed7MZlcOWT3iIXIhI7CsrifLUTJg6CLSvg6rHQfkDQFYlIQBSURdm3AyZkwLbVcO1EOK1v0BWJSIAUlIV9uRXGZ8DOtTD4aWj3vaArEpGARX4eZbnmUO7ZDGMvg53rvP1tFJIiQgxalGWeQ7k7B8b1h72fe9vJtj4vCdWJSCqIfFBCGeZQ7vrUC8n9u+D6GdCqR/KKE5HQi0VQlmjnWhjbHw7thcyZ0Lxb0BWJSMjEOyi3f+y1JHMPQtYcOLFz0BWJSAjFNyi3roJxAwAHw+fC8R2CrkhEQiryo95F2rLCG902g+HzFJIiUqL4BeWmd2Ds5ZBWA4bPh6anBV2RiIRcvIIyZymMy4Aa9SB7PjQ5JeiKRCQF+BqUZtbHzFab2Rozu6uI183MHs1//T0z6+pbMZ++CeOvgNoNIXseNGrj21uJSLT4FpRmlgaMAPoC7YEhZta+0GF9gXb5HzcDj/tRS/uD78LEK6Hu8ZC9ABpUcF1KEYklP1uU3YE1zrm1zrlDwBQgo9AxGcB451kENDCzExNZRMeDb3PXzruhQUvvmmS9kxL55UUkBvwMyubAZwUe5+Q/V95jMLObzWypmS3dtm1b2Ss4vJ+f7n2EL2q1gKy5XotSRKSc/JxHWdQm164Cx+CcGw2MBkhPT//W68WqVou6P5hB3fotoHajMv8zEZGC/AzKHKBlgcctgE0VOKZyTuyU0C8nIvHjZ9d7CdDOzNqYWXVgMDC70DGzgcz80e+zgd3Ouc0+1iQiUm6+tSidc7lmdjvwApAGjHHOfWBmt+S/PgqYD/QD1gBfAdl+1SMiUlG+3uvtnJuPF4YFnxtV4HMH3OZnDSIilRWvO3NERCpAQSkiUgoFpYhIKRSUIiKlUFCKiJRCQSkiUgoFpYhIKcybypg6zGwb8Gk5/1kTYLsP5QQhKucSlfMAnUtYlfdcTnbONS3qhZQLyoows6XOufSg60iEqJxLVM4DdC5hlchzUddbRKQUCkoRkVLEJShHB11AAkXlXKJyHqBzCauEnUssrlGKiFRGXFqUIiIVFqmgDNX2uJVQhvMYll//e2b2hpl1DqLOsijtXAocd5aZHTGzq5JZX3mU5VzMrJeZLTezD8zslWTXWBZl+P2qb2ZzzOzd/PMI7TqxZjbGzLaa2fvFvJ6Yv3nnXCQ+8BYH/gRoC1QH3gXaFzqmH7AAb6+es4HFQdddwfM4F2iY/3nfMJ5HWc+lwHH/wVu79Kqg667Ez6UBsBJolf+4WdB1V/A8fgP8T/7nTYGdQPWgay/mfC4EugLvF/N6Qv7mo9SiDMX2uAlQ6nk4595wzu3Kf7gIb6+hMCrLzwTgx8A0YGsyiyunspzLUGC6c24DgHMujOdTlvNwQF0zM+A4vKDMTW6ZZeOcW4hXX3ES8jcfpaBM2Pa4AStvjTfg/Y8ZRqWei5k1BwYCowi3svxcTgUamtnLZrbMzDKTVl3ZleU8HgPOwNvobwXwU+dcXnLKS7iE/M37uhVEkiVse9yAlblGM7sILyjP97WiiivLufwNuNM5d8RrwIRWWc6lKtAN6A3UAt40s0XOuY/8Lq4cynIelwLLgYuB7wD/MrNXnXN7fK7NDwn5m49SUIZje9zKK1ONZtYJeBLo65zbkaTayqss55IOTMkPySZAPzPLdc7NTEqFZVfW36/tzrl9wD4zWwh0BsIUlGU5j2zgQedd5FtjZuuA04G3klNiQiXmbz7oi7EJvKhbFVgLtOHri9QdCh1zGd+8sPtW0HVX8Dxa4e1ceW7Q9Vb2XAodP5bwDuaU5edyBvBi/rG1gfeB7wZdewXO43Hg3vzPjwc2Ak2Crr2Ec2pN8YM5Cfmbj0yL0kVke9wynsfdQGNgZH5LLNeFcCGDMp5LSijLuTjnVpnZ88B7QB7wpHOuyGkrQSnjz+SPwFgzW4EXMHc650K5opCZPQ30ApqYWQ5wD1ANEvs3rztzRERKEaVRbxERXygoRURKoaAUESmFglJEpBQKShGRUigoJWXlrza0vMBH6/zVe3ab2TtmtsrM7sk/tuDzH5rZw0HXL6kjMvMoJZb2O+fOLPiEmbUGXnXOXW5mdYDlZjY3/+Wjz9cC3jGzGc6515NbsqQitSglspx3K+EyvPuVCz6/H+9e5rAtiCIhpaCUVFarQLd7RuEXzawx3m1rHxR6viHQDliYnDIl1anrLansW13vfBeY2Tt4txE+mH+LXq/8598DTst/fkvSKpWUpqCUKHrVOXd5cc+b2anAa/nXKJcnuTZJQep6S+w4b33IPwF3Bl2LpAYFpcTVKOBCM2sTdCESflo9SESkFGpRioiUQkEpIlIKBaWISCkUlCIipVBQioiUQkEpIlIKBaWISCkUlCIipfh/lgI6h1hNo5EAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 360x360 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.figure(figsize=(5,5))\n",
"plt.plot(fpr, tpr, label=\"model\")\n",
"plt.plot([0, 1], [0,1], label=\"random\")\n",
"plt.xlabel(\"FPR\")\n",
"plt.ylabel(\"TPR\")\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"id": "ab48cd56-e124-45ed-a5d6-77ddd9d83545",
"metadata": {},
"source": [
"* The differences occur, because our evaluation was only for 101 thresholds and sklearn is more accurate"
]
},
{
"cell_type": "markdown",
"id": "93ce16f8-f51e-4dfe-9498-8edf09dcffcb",
"metadata": {},
"source": [
"## ROC AUC\n",
"\n",
"* Area under the ROC curve - useful metric for binary classification\n",
" * For the random model, thes area under the curve is AUC=0.5\n",
" * For the ideal model, thes area under the curve is AUC=1\n",
" * I.e. for all models AUC is between 0.5 and 1\n",
"* Getting the average prediction and the spread within predictions\n",
"* We can use sklearn to calculate AUC"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "5d993a0c-b4cd-4b5f-8d0e-95c60deeebe2",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import auc"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "f62b6fdd-b1dc-4cb7-a9f6-1b0aedc61dc1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8438099868820242"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# for the scores calculted by sklearn\n",
"auc(fpr, tpr)"
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "a390ec4a-a3ba-4fb4-8d3b-f0b0d9d02888",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8438391098010017"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# for our manually calculates scores\n",
"auc(df_scores.fpr, df_scores.tpr)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "f8348150-4153-4a28-a623-d7d89992fd1d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9999430203759136"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# for the ideal model\n",
"auc(df_ideal.fpr, df_ideal.tpr)"
]
},
{
"cell_type": "markdown",
"id": "0d2da908-0fb6-46f7-baf4-1fb604c003dc",
"metadata": {},
"source": [
"* The differences again are due to the lower precission of our manually calculations\n",
"* This calculation can also be done in one step, directly from the validations and predictions"
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "d0ef035d-a7fc-4ab4-b98f-1ac3388a6e2a",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import roc_auc_score"
]
},
{
"cell_type": "code",
"execution_count": 58,
"id": "a604126e-541b-4892-a1ce-ef59a056c01b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8438099868820242"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"roc_auc_score(y_val, y_pred)"
]
},
{
"cell_type": "markdown",
"id": "7ff55088-fa09-4400-be34-b3010fba2bd9",
"metadata": {},
"source": [
"* The above line is a shortcut for:"
]
},
{
"cell_type": "code",
"execution_count": 59,
"id": "4f9043a6-9c6a-4233-8478-42e68e3d8f39",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8438099868820242"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fpr, tpr, thresholds = roc_curve(y_val, y_pred)\n",
"auc(fpr, tpr)"
]
},
{
"cell_type": "markdown",
"id": "442adb06-b440-4306-8df8-6b5b0229c3d4",
"metadata": {},
"source": [
"**AUC Interpretation**\n",
"\n",
"* AUC tells us, what the propabilty is of a randomly selected positive sample has a higher score than a randomly selected negative sample"
]
},
{
"cell_type": "code",
"execution_count": 60,
"id": "bacd7007-5dbf-44d2-a08c-999082bb0745",
"metadata": {},
"outputs": [],
"source": [
"# scores for negative samples\n",
"neg = y_pred[y_val==0]\n",
"# scores for positive samples\n",
"pos = y_pred[y_val==1]"
]
},
{
"cell_type": "code",
"execution_count": 61,
"id": "b00c5d9b-633e-46c9-86fb-25bf845ce664",
"metadata": {},
"outputs": [],
"source": [
"# randomly select a positive sample\n",
"import random\n",
"\n",
"pos_ind = random.randint(0, len(pos) -1)\n",
"neg_ind = random.randint(0, len(neg) -1)"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "bc3579ac-3be3-4b4d-8b03-a555c5fac729",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# compare the scores of the positive and negative sample\n",
"pos[pos_ind] > neg[neg_ind]"
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "b9e20165-74f5-409e-a32f-c84e682fa6de",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8397"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Do this many times ...\n",
"n = 10000\n",
"success = 0\n",
"for i in range(n):\n",
" pos_ind = random.randint(0, len(pos) -1)\n",
" neg_ind = random.randint(0, len(neg) -1)\n",
" \n",
" if pos[pos_ind] > neg[neg_ind]:\n",
" success += 1\n",
" \n",
"success/n"
]
},
{
"cell_type": "markdown",
"id": "a469d591-d4e1-476d-b007-1133be3e0ec3",
"metadata": {},
"source": [
"* Our result is pretty close to the previous calculated AUC value"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "1cfc4110-63a1-40b1-a5d9-bae7d21d376d",
"metadata": {},
"outputs": [],
"source": [
"# alternative vectorized implementation\n",
"n = 50000\n",
"pos_ind = np.random.randint(0, len(pos), size=n)\n",
"neg_ind = np.random.randint(0, len(neg), size=n)"
]
},
{
"cell_type": "code",
"execution_count": 65,
"id": "514be844-29ef-41e8-9b70-8b0e95bf2b8e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.84484"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(pos[pos_ind] > neg[neg_ind]).mean()"
]
},
{
"cell_type": "markdown",
"id": "acbd1364-4c2f-4782-8747-09cdba32af77",
"metadata": {
"tags": []
},
"source": [
"## Cross Validation\n",
"\n",
"* Evaluating the same model on different subsets of data\n",
"* Getting the average prediction and the spread within predictions"
]
},
{
"cell_type": "code",
"execution_count": 101,
"id": "d52bd146-1126-4145-b609-5762483f1fd8",
"metadata": {},
"outputs": [],
"source": [
"# train a model\n",
"def train(df, y_train):\n",
" dicts = df[categorical + numerical].to_dict(orient=\"records\")\n",
" dv = DictVectorizer(sparse=False)\n",
" X_train = dv.fit_transform(dicts)\n",
" \n",
" model = LogisticRegression(solver=\"liblinear\")\n",
" model.fit(X_train, y_train)\n",
" \n",
" return dv, model"
]
},
{
"cell_type": "code",
"execution_count": 102,
"id": "a6a000a6-73b4-4de1-98b7-88a2b14be957",
"metadata": {},
"outputs": [],
"source": [
"dv, model = train(df_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 103,
"id": "66e2a623-bb7d-4014-a785-e8520d85ed5e",
"metadata": {},
"outputs": [],
"source": [
"# make predictions \n",
"def predict(df, dv, model):\n",
" dicts = df[categorical + numerical].to_dict(orient=\"records\")\n",
" X = dv.transform(dicts)\n",
" y_pred = model.predict_proba(X)[:,1]\n",
" return y_pred"
]
},
{
"cell_type": "markdown",
"id": "cc46c450-541f-464d-906c-793827bce60d",
"metadata": {},
"source": [
"* To do the k-fold split we can use sklearn"
]
},
{
"cell_type": "code",
"execution_count": 104,
"id": "39a4bd14-12e1-468a-8269-b2722f66a529",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: tqdm in /home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages (4.63.0)\n"
]
}
],
"source": [
"!pip install tqdm"
]
},
{
"cell_type": "code",
"execution_count": 105,
"id": "eee40d6f-fb02-4b7a-8453-932667ff90e0",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import KFold"
]
},
{
"cell_type": "code",
"execution_count": 106,
"id": "99e898c4-5a9f-4250-86aa-dd998ce84167",
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm"
]
},
{
"cell_type": "code",
"execution_count": 107,
"id": "ca9bcfc2-b4c6-47fb-a60a-b736242653be",
"metadata": {},
"outputs": [],
"source": [
"kfold = KFold(n_splits=10, shuffle=True, random_state=1)"
]
},
{
"cell_type": "markdown",
"id": "b267a65b-6c06-4619-b5e3-03d02a575d89",
"metadata": {},
"source": [
"* ```kfold.split(df_train_full)``` generates an iterator with indices for training and validation"
]
},
{
"cell_type": "code",
"execution_count": 108,
"id": "3813e526-d414-42ea-9636-8fb8e8387587",
"metadata": {},
"outputs": [],
"source": [
"train_idx, val_idx = next(kfold.split(df_train_full))"
]
},
{
"cell_type": "code",
"execution_count": 109,
"id": "c10e0edc-c08d-45b8-8992-79deade64ee0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"full data length: 5634, train data length: 5070, validation data length 564\n"
]
}
],
"source": [
"print(f\"full data length: {len(df_train_full)}, train data length: {len(train_idx)}, validation data length {len(val_idx)}\")"
]
},
{
"cell_type": "code",
"execution_count": 110,
"id": "4870de48-1af3-4ab2-9873-0f177a757a5f",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"10it [00:02, 4.19it/s]\n"
]
}
],
"source": [
"# Loop through different folds\n",
"scores = []\n",
"for train_idx, val_idx in tqdm(kfold.split(df_train_full)):\n",
" df_train = df_train_full.iloc[train_idx]\n",
" df_val = df_train_full.iloc[val_idx]\n",
" \n",
" y_train = df_train.churn.values\n",
" y_val = df_val.churn.values\n",
" \n",
" dv, model = train(df_train, y_train)\n",
" y_pred = predict(df_val, dv, model)\n",
" \n",
" roc_auc = roc_auc_score(y_val, y_pred)\n",
" scores.append(roc_auc)"
]
},
{
"cell_type": "code",
"execution_count": 111,
"id": "deed409d-79ea-4fdb-a521-7d873246a6bc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0.8493392490816277,\n",
" 0.8413366336633662,\n",
" 0.8590269587894291,\n",
" 0.8330260883877869,\n",
" 0.8242710918114144,\n",
" 0.8416250416250417,\n",
" 0.8437154021491371,\n",
" 0.8223355471220746,\n",
" 0.8450570623981029,\n",
" 0.8611811367685119]"
]
},
"execution_count": 111,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scores"
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "ea898a5e-9a02-4632-981b-e26a678b74e0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mean auc: 0.842, spread: 1.23%\n"
]
}
],
"source": [
"print(f\"mean auc: {np.mean(scores):.3f}, spread: {np.std(scores)*100:.3}%\")"
]
},
{
"cell_type": "markdown",
"id": "2d5aa5c9-cde0-447d-aaa2-7353b5584e51",
"metadata": {},
"source": [
"* In logisgtic regression a regularization parameter \"C\" can be included\n",
"* We can then test different regularization paramters"
]
},
{
"cell_type": "code",
"execution_count": 125,
"id": "81e7e6a1-b9f0-43c3-81e0-a80362a67987",
"metadata": {},
"outputs": [],
"source": [
"# train a model\n",
"def train2(df, y_train, C=1.0):\n",
" dicts = df[categorical + numerical].to_dict(orient=\"records\")\n",
" dv = DictVectorizer(sparse=False)\n",
" X_train = dv.fit_transform(dicts)\n",
" \n",
" model = LogisticRegression(solver=\"liblinear\", C=C)#, max_iter=10000)\n",
" model.fit(X_train, y_train)\n",
" \n",
" return dv, model"
]
},
{
"cell_type": "code",
"execution_count": 135,
"id": "2739be4e-794f-42d2-8140-f83e03583bb6",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 4.98it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 0.001\n",
"mean auc: 0.825, spread: 1.31%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 4.58it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 0.01\n",
"mean auc: 0.839, spread: 0.872%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 4.75it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 0.1\n",
"mean auc: 0.841, spread: 0.748%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 3.69it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 0.5\n",
"mean auc: 0.841, spread: 0.74%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 3.21it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 1\n",
"mean auc: 0.841, spread: 0.739%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 5/5 [00:01<00:00, 3.95it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Regularization parameter C: 10\n",
"mean auc: 0.841, spread: 0.745%\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"# Loop through different folds\n",
"n_splits = 5\n",
"\n",
"for C in [0.001, 0.01, 0.1, 0.5, 1, 10]: \n",
" scores = []\n",
" \n",
" kfold = KFold(n_splits=n_splits, shuffle=True, random_state=1)\n",
" for train_idx, val_idx in tqdm(kfold.split(df_train_full), total=n_splits):\n",
" \n",
" df_train = df_train_full.iloc[train_idx]\n",
" df_val = df_train_full.iloc[val_idx]\n",
" \n",
" y_train = df_train.churn.values\n",
" y_val = df_val.churn.values\n",
" \n",
" dv, model = train2(df_train, y_train, C=C)\n",
" y_pred = predict(df_val, dv, model)\n",
" \n",
" roc_auc = roc_auc_score(y_val, y_pred)\n",
" scores.append(roc_auc)\n",
" \n",
" print(f\"Regularization parameter C: {C}\") \n",
" print(f\"mean auc: {np.mean(scores):.3f}, spread: {np.std(scores)*100:.3}%\")"
]
},
{
"cell_type": "markdown",
"id": "f52f0065-c48a-4f2f-8361-8a56927a454f",
"metadata": {},
"source": [
"* Train the final model"
]
},
{
"cell_type": "code",
"execution_count": 136,
"id": "c3cea9ed-c35c-4b0f-8e5c-f292164cd2cc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8579400803839363"
]
},
"execution_count": 136,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dv, model = train2(df_train_full, df_train_full.churn.values, C=1)\n",
"y_pred = predict(df_test, dv, model)\n",
" \n",
"roc_auc = roc_auc_score(y_test, y_pred)\n",
"roc_auc"
]
},
{
"cell_type": "markdown",
"id": "42ae1539-1f7a-4c9f-87fe-65d381f146b2",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"* Metric: A single number that describes the performance of a model\n",
"* Accuracy: fraction of correct answers; sometimes misleading\n",
"* Precision and Recall are less misleading when we have class imbalance\n",
"* ROC curve: A way to evaluate the performance at all thresholds; ok to use with class imbalance\n",
"* K-fold Cross Validation: more reliable estimate for performance (mean + std)"
]
},
{
"cell_type": "markdown",
"id": "a0e291be-866a-4404-8ce5-9a2ed45b4d9a",
"metadata": {},
"source": [
"# Explore More\n",
"\n",
"* Check presicion and recall for the dummy model\n",
"* F1 score = 2 * P * R/(P + R)\n",
"* evaluate precision and recall at different thresholds, plot P vs R - this way you will get the precision recall curve (similar ti ROC curve)\n",
"* Area under the PR curve is also a useful metric\n",
"\n",
"Other projects\n",
"\n",
"* calculate the metrics for the datasets from the previous week\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d2b83b0f-6670-48a4-9277-317004101ebf",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "ml-zoomcamp",
"language": "python",
"name": "ml-zoomcamp"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}