evaluation/Classification_Analysis_Ablation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calibrated Explanations for Binary Classification\n",
    "## Ablatioon Analysis\n",
    "\n",
    "Author: Tuwe Löfström (tuwe.lofstrom@ju.se)  \n",
    "Copyright 2023 Tuwe Löfström  \n",
    "License: BSD 3 clause\n",
    "Sources:\n",
    "1. [\"Calibrated Explanations: with Uncertainty Information and Counterfactuals\"](https://arxiv.org/abt/2305.02305) by [Helena Löfström](https://github.com/Moffran), [Tuwe Löfström](https://github.com/tuvelofstrom), Ulf Johansson, and Cecilia Sönströd.\n",
    "\n",
    "### 1. Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats as st"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 Import results from the pickled result file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('results_ablation.pkl', 'rb') as f:\n",
    "    results = pickle.load(f)\n",
    "data_characteristics = {'colic': 60, \n",
    "                        'creditA': 43, \n",
    "                        'diabetes': 9, \n",
    "                        'german': 28, \n",
    "                        'haberman': 4, \n",
    "                        'haberman': 4,\n",
    "                        'heartC': 23,\n",
    "                        'heartH': 21,\n",
    "                        'heartS': 14,\n",
    "                        'hepati': 20,\n",
    "                        'iono': 34,\n",
    "                        'je4042': 9,\n",
    "                        'je4243': 9, \n",
    "                        'kc1': 22,\n",
    "                        'kc2': 22,\n",
    "                        'kc3': 40,\n",
    "                        'liver': 7,\n",
    "                        'pc1req': 9,\n",
    "                        'pc4': 38,\n",
    "                        'sonar': 61,\n",
    "                        'spect': 23,\n",
    "                        'spectf': 45,\n",
    "                        'transfusion': 5,\n",
    "                        'ttt': 28,\n",
    "                        'vote': 17,\n",
    "                        'wbc': 10,}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 Ablation analysis\n",
    "The ablation analysis is focused on evaluating how the algorithm is affected by the calibration size and the number of percentiles sampled for numerical features. It is using a similar setup as the stability experiment, but with the following changes:\n",
    "* The number of percentiles sampled for numerical features is varied between 1, 2, 3 (default), 4, and 9. The set of percentiles used are: [50], [33, 67], [25, 50, 75], [20, 40, 60, 80], [10, 20, 30, 40, 50, 60, 70, 80, 90]\n",
    "* The calibration size is varied between 10%, 20% and 40% of the data not used for testing.\n",
    "* Test size is fixed to 10% of the data. \n",
    "* Only one repetition per percentile and calibration size is used.\n",
    "\n",
    "Everything was run on 25 datasets. See the `Classification_Experiment_Ablation.py` for details on the experiment.\n",
    "\n",
    "The tabulated results are the mean variance of the ablation measured per calibration size or percentile sampling. The variance is measured per instance and computed over the runs having the same calibration size/percentile sampling on the feature importance weight of the most influential feature, defined as the feature most often having highest absolute feature importance weight. The average variance is computed over the entire test set. The most influential feature is used since it is the feature that is most likely to be used in a decision but also the feature with the greatest expected variation (as a consequence of the weights having the highest absolute values). \n",
    "\n",
    "#### 3.1 Calibration Size\n",
    "First out is a table with results per calibration size. Since different sampling sizes may result in different results for numerical features, the mean variance is only expected to be 0 for categorical-only datasets. The results are printed as a latex table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " & xGB & xGB & xGB & xGB & xGB & xGB & RF & RF & RF & RF & RF & RF \\\\\n",
      "Calibration Size & 0.1 & 0.2 & 0.4 & 0.1 & 0.2 & 0.4 & 0.1 & 0.2 & 0.4 & 0.1 & 0.2 & 0.4\\\\\n",
      "Dataset & CE & CE & CE & CCE & CCE & CCE & CE & CE & CE & CCE & CCE & CCE \\\\\n",
      "\\hline\n",
      "colic & 2.4e-05 & 1.8e-05 & 2.7e-05 & 2.4e-05 & 1.8e-05 & 2.7e-05 & 6.9e-08 & 1.1e-06 & 7.4e-07 & 6.9e-08 & 1.1e-06 & 7.4e-07 \\\\\n",
      "creditA & 0.00056 & 0.00043 & 0.0004 & 0.00056 & 0.00043 & 0.0004 & 2.4e-05 & 4.1e-05 & 6e-05 & 2.4e-05 & 4.1e-05 & 6e-05 \\\\\n",
      "diabetes & 0.00024 & 0.00024 & 0.00027 & 0.00024 & 0.00024 & 0.00027 & 0.00037 & 0.00032 & 0.0003 & 0.00037 & 0.00032 & 0.0003 \\\\\n",
      "german & 2.2e-05 & 1.1e-05 & 2.2e-05 & 2.2e-05 & 1.1e-05 & 2.2e-05 & 5.1e-06 & 2.6e-06 & 1.8e-05 & 5.1e-06 & 2.6e-06 & 1.8e-05 \\\\\n",
      "haberman & 0.0014 & 0.00084 & 0.00062 & 0.0014 & 0.00084 & 0.00062 & 0.00034 & 0.00094 & 0.00083 & 0.00034 & 0.00094 & 0.00083 \\\\\n",
      "heartC & 7.6e-06 & 6.4e-05 & 4.7e-05 & 7.6e-06 & 6.4e-05 & 4.7e-05 & 2.7e-07 & 1.5e-05 & 1.4e-05 & 2.7e-07 & 1.5e-05 & 1.4e-05 \\\\\n",
      "heartH & 5.2e-05 & 3.2e-05 & 5.6e-05 & 5.2e-05 & 3.2e-05 & 5.6e-05 & 2.3e-06 & 1.3e-06 & 9e-07 & 2.3e-06 & 1.3e-06 & 9e-07 \\\\\n",
      "heartS & 0.00015 & 0.00026 & 0.00025 & 0.00015 & 0.00026 & 0.00025 & 3e-05 & 2.7e-05 & 5.5e-05 & 3e-05 & 2.7e-05 & 5.5e-05 \\\\\n",
      "hepati & 9.5e-05 & 0.00017 & 0.00014 & 9.5e-05 & 0.00017 & 0.00014 & 2e-05 & 4.7e-05 & 6.6e-05 & 2e-05 & 4.7e-05 & 6.6e-05 \\\\\n",
      "iono & 0.00011 & 0.00011 & 7.5e-05 & 0.00011 & 0.00011 & 7.5e-05 & 5.2e-05 & 5.2e-05 & 8.2e-05 & 5.2e-05 & 5.2e-05 & 8.2e-05 \\\\\n",
      "je4042 & 0.00016 & 0.00022 & 0.00026 & 0.00016 & 0.00022 & 0.00026 & 3.5e-05 & 5e-05 & 4.5e-05 & 3.5e-05 & 5e-05 & 4.5e-05 \\\\\n",
      "je4243 & 4.8e-05 & 5.9e-05 & 5.1e-05 & 4.8e-05 & 5.9e-05 & 5.1e-05 & 0.00036 & 0.00026 & 0.00024 & 0.00036 & 0.00026 & 0.00024 \\\\\n",
      "kc1 & 0.00025 & 0.00017 & 0.00016 & 0.00025 & 0.00017 & 0.00016 & 0.00037 & 0.00022 & 0.00017 & 0.00037 & 0.00022 & 0.00017 \\\\\n",
      "kc2 & 0.00019 & 0.0002 & 0.00016 & 0.00019 & 0.0002 & 0.00016 & 5.6e-06 & 3.4e-05 & 8.8e-05 & 5.6e-06 & 3.4e-05 & 8.8e-05 \\\\\n",
      "kc3 & 5e-06 & 9e-06 & 7.2e-06 & 5e-06 & 9e-06 & 7.2e-06 & 7.3e-06 & 6.3e-06 & 4.5e-06 & 7.3e-06 & 6.3e-06 & 4.5e-06 \\\\\n",
      "liver & 0.00014 & 0.00013 & 0.0001 & 0.00014 & 0.00013 & 0.0001 & 0.00061 & 0.00044 & 0.0005 & 0.00061 & 0.00044 & 0.0005 \\\\\n",
      "pc1req & 0.0 & 3.5e-35 & 2.9e-35 & 0.0 & 3.5e-35 & 2.9e-35 & 6.3e-07 & 3.1e-07 & 2.1e-07 & 6.3e-07 & 3.1e-07 & 2.1e-07 \\\\\n",
      "pc4 & 0.00014 & 0.00011 & 9.9e-05 & 0.00014 & 0.00011 & 9.9e-05 & 4.1e-05 & 4.7e-05 & 4.8e-05 & 4.1e-05 & 4.7e-05 & 4.8e-05 \\\\\n",
      "sonar & 1.3e-05 & 0.00015 & 0.0002 & 1.3e-05 & 0.00015 & 0.0002 & 4.8e-05 & 0.00013 & 0.00011 & 4.8e-05 & 0.00013 & 0.00011 \\\\\n",
      "spect & 1.4e-34 & 1.1e-34 & 7e-35 & 1.4e-34 & 1.1e-34 & 7e-35 & 0.0 & 1.8e-35 & 2.3e-35 & 0.0 & 1.8e-35 & 2.3e-35 \\\\\n",
      "spectf & 0.00014 & 7e-05 & 8.5e-05 & 0.00014 & 7e-05 & 8.5e-05 & 8.5e-05 & 5.9e-05 & 7.7e-05 & 8.5e-05 & 5.9e-05 & 7.7e-05 \\\\\n",
      "transfusion & 0.00034 & 0.00025 & 0.00025 & 0.00034 & 0.00025 & 0.00025 & 0.00031 & 0.00035 & 0.00033 & 0.00031 & 0.00035 & 0.00033 \\\\\n",
      "ttt & 2.8e-33 & 3.2e-33 & 3.3e-33 & 2.8e-33 & 3.2e-33 & 3.3e-33 & 1e-33 & 1.6e-33 & 1.7e-33 & 1e-33 & 1.6e-33 & 1.7e-33 \\\\\n",
      "vote & 1.3e-34 & 1.7e-34 & 1.2e-34 & 1.3e-34 & 1.7e-34 & 1.2e-34 & 1.3e-34 & 8.9e-35 & 1.1e-34 & 1.3e-34 & 8.9e-35 & 1.1e-34 \\\\\n",
      "wbc & 0.00016 & 0.00027 & 0.00036 & 0.00016 & 0.00027 & 0.00036 & 0.00056 & 0.00077 & 0.00075 & 0.00056 & 0.00077 & 0.00075 \\\\\n",
      "\\hline\n",
      "Average & 0.00017 & 0.00015 & 0.00015 & 0.00017 & 0.00015 & 0.00015 & 0.00013 & 0.00015 & 0.00015 & 0.00013 & 0.00015 & 0.00015 \\\\\n"
     ]
    }
   ],
   "source": [
    "ranking = {}\n",
    "val = {}\n",
    "average_results = {}\n",
    "n = results['test_size']\n",
    "cal_sizes = results['calibration_sizes']\n",
    "perc_samples = results['sample_percentiles']\n",
    "\n",
    "for a in ['xGB', 'RF']:\n",
    "    for cal in cal_sizes:\n",
    "        average_results[a+'_'+str(cal)+'_ce'] = []\n",
    "        average_results[a+'_'+str(cal)+'_cce'] = []\n",
    "\n",
    "print(' & xGB & xGB & xGB & xGB & xGB & xGB & RF & RF & RF & RF & RF & RF \\\\\\\\\\nCalibration Size', end='')\n",
    "for i in range(2):\n",
    "    for key in ['ce', 'cce']:  \n",
    "        for cal in cal_sizes:\n",
    "            print(f' & {cal}',end='')\n",
    "print('\\\\\\\\')\n",
    "print('Dataset & CE & CE & CE & CCE & CCE & CCE & CE & CE & CE & CCE & CCE & CCE \\\\\\\\\\n\\\\hline')\n",
    "for d in np.sort([k for k in results.keys()]):\n",
    "    if d in ['test_size', 'calibration_sizes', 'sample_percentiles']:\n",
    "        continue\n",
    "    print(d, end='')\n",
    "    for a in results[d]:\n",
    "        ablation = results[d][a]['ablation']\n",
    "        \n",
    "        for key in ['ce', 'cce']:    \n",
    "            n = len(ablation[key][cal_sizes[0]][str(perc_samples[0])][0])\n",
    "            ranks = []\n",
    "            values = []\n",
    "            for j in range(n):\n",
    "                rank = []\n",
    "                value = []\n",
    "                for cal in ablation[key]:\n",
    "                    for p in ablation[key][cal]:\n",
    "                        rank.append(np.argsort(np.abs(ablation[key][cal][p][0][j]['predict']))[-1:][0])\n",
    "                ranks.append(rank)\n",
    "                values.append(value)\n",
    "            ranking[key] = st.mode(ranks, axis=1)[0] # Find most important feature per instance\n",
    "            # print(ranking[key])\n",
    "            \n",
    "            value = []\n",
    "            for cal in ablation[key]: \n",
    "                print(' & ', end='')  \n",
    "                for j in range(n):\n",
    "                    values = [ablation[key][cal][p][0][j]['predict'][ranking[key][j]] for p in ablation[key][cal]]\n",
    "                    value.append([np.mean(values), np.var(values)])\n",
    "                val[key] = value \n",
    "\n",
    "                res = np.mean([t[1] for t in val[key]])\n",
    "                average_results[a+'_'+str(cal)+'_'+key].append(res)\n",
    "                print(f'{res:.2}',end='')\n",
    "    print(' \\\\\\\\')\n",
    "print('\\\\hline\\nAverage', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']:  \n",
    "        for cal in ablation[key]:\n",
    "            print(' & ', end='')\n",
    "            print(f'{np.mean(average_results[a+\"_\"+str(cal)+\"_\"+key]):.2}',end='')\n",
    "print(' \\\\\\\\')\n",
    "# df = pd.DataFrame.from_dict(average_results, orient='index')\n",
    "# display (df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most interesting observation from the results above is that difference in mean variance is fairly low between the different calibration sizes. This indicates that the calibration size does not have a large impact on the feature importance weights. In fact, a smaller calibration set even tend to have a lower mean variance.\n",
    "#### 3.2 Percentile Sampling\n",
    "Below are the results per percentile sampling. The results are printed as a latex table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF \\\\\n",
      "Sample Size & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9\\\\\n",
      "Dataset & CE & CE & CE & CE & CE & CCE & CCE & CCE & CCE & CCE & CE & CE & CE & CE & CE & CCE & CCE & CCE & CCE & CCE \\\\\n",
      "\\hline\n",
      "colic & 0.0026 & 0.0027 & 0.0027 & 0.0027 & 0.0027 & 0.0026 & 0.0027 & 0.0027 & 0.0027 & 0.0027 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 \\\\\n",
      "creditA & 0.007 & 0.0066 & 0.0064 & 0.0062 & 0.0061 & 0.007 & 0.0066 & 0.0064 & 0.0062 & 0.0061 & 0.003 & 0.003 & 0.003 & 0.003 & 0.0029 & 0.003 & 0.003 & 0.003 & 0.003 & 0.0029 \\\\\n",
      "diabetes & 0.0045 & 0.0044 & 0.0043 & 0.0042 & 0.0041 & 0.0045 & 0.0044 & 0.0043 & 0.0042 & 0.0041 & 0.0096 & 0.0097 & 0.0096 & 0.0095 & 0.0094 & 0.0096 & 0.0097 & 0.0096 & 0.0095 & 0.0094 \\\\\n",
      "german & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0011 & 0.0027 & 0.0027 & 0.0026 & 0.0026 & 0.0026 & 0.0027 & 0.0027 & 0.0026 & 0.0026 & 0.0026 \\\\\n",
      "haberman & 0.012 & 0.012 & 0.012 & 0.011 & 0.011 & 0.012 & 0.012 & 0.012 & 0.011 & 0.011 & 0.0072 & 0.0064 & 0.0061 & 0.0057 & 0.0055 & 0.0072 & 0.0064 & 0.0061 & 0.0057 & 0.0055 \\\\\n",
      "heartC & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.0043 & 0.004 & 0.004 & 0.004 & 0.004 & 0.0041 & 0.004 & 0.004 & 0.004 & 0.004 & 0.0041 \\\\\n",
      "heartH & 0.0054 & 0.0054 & 0.0054 & 0.0053 & 0.0053 & 0.0054 & 0.0054 & 0.0054 & 0.0053 & 0.0053 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 & 0.0069 \\\\\n",
      "heartS & 0.0096 & 0.0087 & 0.0084 & 0.0083 & 0.0082 & 0.0096 & 0.0087 & 0.0084 & 0.0083 & 0.0082 & 0.0054 & 0.0053 & 0.0052 & 0.0052 & 0.0052 & 0.0054 & 0.0053 & 0.0052 & 0.0052 & 0.0052 \\\\\n",
      "hepati & 0.0075 & 0.0067 & 0.0065 & 0.0063 & 0.0061 & 0.0075 & 0.0067 & 0.0065 & 0.0063 & 0.0061 & 0.001 & 0.0011 & 0.0012 & 0.0011 & 0.0011 & 0.001 & 0.0011 & 0.0012 & 0.0011 & 0.0011 \\\\\n",
      "iono & 0.002 & 0.002 & 0.002 & 0.002 & 0.0021 & 0.002 & 0.002 & 0.002 & 0.002 & 0.0021 & 0.0041 & 0.0041 & 0.0041 & 0.0041 & 0.0039 & 0.0041 & 0.0041 & 0.0041 & 0.0041 & 0.0039 \\\\\n",
      "je4042 & 0.0087 & 0.0087 & 0.0083 & 0.0082 & 0.008 & 0.0087 & 0.0087 & 0.0083 & 0.0082 & 0.008 & 0.0054 & 0.0052 & 0.0052 & 0.0052 & 0.0051 & 0.0054 & 0.0052 & 0.0052 & 0.0052 & 0.0051 \\\\\n",
      "je4243 & 0.0027 & 0.0027 & 0.0026 & 0.0026 & 0.0026 & 0.0027 & 0.0027 & 0.0026 & 0.0026 & 0.0026 & 0.003 & 0.0028 & 0.0027 & 0.0026 & 0.0025 & 0.003 & 0.0028 & 0.0027 & 0.0026 & 0.0025 \\\\\n",
      "kc1 & 0.0064 & 0.0064 & 0.0063 & 0.0062 & 0.0061 & 0.0064 & 0.0064 & 0.0063 & 0.0062 & 0.0061 & 0.0048 & 0.0041 & 0.004 & 0.0038 & 0.0037 & 0.0048 & 0.0041 & 0.004 & 0.0038 & 0.0037 \\\\\n",
      "kc2 & 0.0045 & 0.0043 & 0.0042 & 0.0042 & 0.0042 & 0.0045 & 0.0043 & 0.0042 & 0.0042 & 0.0042 & 0.0039 & 0.0038 & 0.0038 & 0.0037 & 0.0037 & 0.0039 & 0.0038 & 0.0038 & 0.0037 & 0.0037 \\\\\n",
      "kc3 & 0.00065 & 0.00063 & 0.00063 & 0.00063 & 0.00062 & 0.00065 & 0.00063 & 0.00063 & 0.00063 & 0.00062 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 \\\\\n",
      "liver & 0.0097 & 0.0098 & 0.0098 & 0.0098 & 0.0097 & 0.0097 & 0.0098 & 0.0098 & 0.0098 & 0.0097 & 0.0057 & 0.0059 & 0.0057 & 0.0057 & 0.0056 & 0.0057 & 0.0059 & 0.0057 & 0.0057 & 0.0056 \\\\\n",
      "pc1req & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.0035 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 & 0.004 \\\\\n",
      "pc4 & 0.00097 & 0.00098 & 0.001 & 0.001 & 0.001 & 0.00097 & 0.00098 & 0.001 & 0.001 & 0.001 & 0.00085 & 0.00082 & 0.00081 & 0.00081 & 0.00081 & 0.00085 & 0.00082 & 0.00081 & 0.00081 & 0.00081 \\\\\n",
      "sonar & 0.0095 & 0.0087 & 0.0084 & 0.0084 & 0.0083 & 0.0095 & 0.0087 & 0.0084 & 0.0084 & 0.0083 & 0.0048 & 0.0048 & 0.0047 & 0.0048 & 0.0048 & 0.0048 & 0.0048 & 0.0047 & 0.0048 & 0.0048 \\\\\n",
      "spect & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 & 0.0021 \\\\\n",
      "spectf & 0.0094 & 0.0097 & 0.0097 & 0.0097 & 0.0096 & 0.0094 & 0.0097 & 0.0097 & 0.0097 & 0.0096 & 0.0058 & 0.0058 & 0.0057 & 0.0056 & 0.0056 & 0.0058 & 0.0058 & 0.0057 & 0.0056 & 0.0056 \\\\\n",
      "transfusion & 0.0011 & 0.001 & 0.001 & 0.00099 & 0.00097 & 0.0011 & 0.001 & 0.001 & 0.00099 & 0.00097 & 0.0041 & 0.0036 & 0.0034 & 0.0033 & 0.0033 & 0.0041 & 0.0036 & 0.0034 & 0.0033 & 0.0033 \\\\\n",
      "ttt & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0037 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 & 0.0017 \\\\\n",
      "vote & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.003 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 & 0.0023 \\\\\n",
      "wbc & 0.0085 & 0.0082 & 0.0082 & 0.0082 & 0.0079 & 0.0085 & 0.0082 & 0.0082 & 0.0082 & 0.0079 & 0.002 & 0.0018 & 0.0016 & 0.0016 & 0.0015 & 0.002 & 0.0018 & 0.0016 & 0.0016 & 0.0015 \\\\\n",
      "\\hline\n",
      "Average & 0.0053 & 0.0051 & 0.005 & 0.005 & 0.0049 & 0.0053 & 0.0051 & 0.005 & 0.005 & 0.0049 & 0.004 & 0.0039 & 0.0039 & 0.0038 & 0.0038 & 0.004 & 0.0039 & 0.0039 & 0.0038 & 0.0038 \\\\\n"
     ]
    }
   ],
   "source": [
    "ranking = {}\n",
    "val = {}\n",
    "average_results = {}\n",
    "n = results['test_size']\n",
    "cal_sizes = results['calibration_sizes']\n",
    "perc_samples = results['sample_percentiles']\n",
    "\n",
    "for a in ['xGB', 'RF']:\n",
    "    for p in perc_samples:\n",
    "        average_results[a+'_'+str(p)+'_ce'] = []\n",
    "        average_results[a+'_'+str(p)+'_cce'] = []\n",
    "\n",
    "print(' & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF \\\\\\\\\\nSample Size', end='')\n",
    "for i in range(2):\n",
    "    for key in ['ce', 'cce']:  \n",
    "        for p in perc_samples:\n",
    "            print(f' & {str(len(p))}',end='')\n",
    "print('\\\\\\\\')\n",
    "print('Dataset & CE & CE & CE & CE & CE & CCE & CCE & CCE & CCE & CCE & CE & CE & CE & CE & CE & CCE & CCE & CCE & CCE & CCE \\\\\\\\\\n\\\\hline')\n",
    "for d in np.sort([k for k in results.keys()]):\n",
    "    if d in ['test_size', 'calibration_sizes', 'sample_percentiles']:\n",
    "        continue\n",
    "    print(d, end='')\n",
    "    for a in results[d]:\n",
    "        ablation = results[d][a]['ablation']\n",
    "        \n",
    "        for key in ['ce', 'cce']:    \n",
    "            n = len(ablation[key][cal_sizes[0]][str(perc_samples[0])][0])\n",
    "            ranks = []\n",
    "            values = []\n",
    "            for j in range(n):\n",
    "                rank = []\n",
    "                value = []\n",
    "                for cal in ablation[key]:\n",
    "                    for p in ablation[key][cal]:\n",
    "                        rank.append(np.argsort(np.abs(ablation[key][cal][p][0][j]['predict']))[-1:][0])\n",
    "                ranks.append(rank)\n",
    "                values.append(value)\n",
    "            ranking[key] = st.mode(ranks, axis=1)[0] # Find most important feature per instance\n",
    "            # print(ranking[key])\n",
    "            \n",
    "            value = []\n",
    "            for p in ablation[key][cal]: \n",
    "                print(' & ', end='')  \n",
    "                for j in range(n):\n",
    "                    values = [ablation[key][cal][p][0][j]['predict'][ranking[key][j]] for cal in ablation[key]]\n",
    "                    value.append([np.mean(values), np.var(values)])\n",
    "                val[key] = value \n",
    "\n",
    "                res = np.mean([t[1] for t in val[key]]) # mean of instance variance\n",
    "                average_results[a+'_'+p+'_'+key].append(res)\n",
    "                print(f'{res:.2}',end='')\n",
    "    print(' \\\\\\\\')\n",
    "print('\\\\hline\\nAverage', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']:  \n",
    "        for p in ablation[key][cal]:\n",
    "            print(' & ', end='')\n",
    "            print(f'{np.mean(average_results[a+\"_\"+p+\"_\"+key]):.2}',end='')\n",
    "print(' \\\\\\\\')\n",
    "# df = pd.DataFrame.from_dict(average_results, orient='index')\n",
    "# display (df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if there is some difference in the mean variance when varying the percentile sampling, the difference can mainly be attributed to the difference in underlying ML algorithm. There is a tendency that a larger set of percentiles tend to reduce the mean variance, which is expected. However, the tendency is not very strong. \n",
    "\n",
    "### 4 Computing time\n",
    "Now, lets look at the runtime taken to compute the explanations. The tabulated runtimes are the average time in seconds per instance. First, detailed results per calibration size AND percentile sampling is shown, then results aggregated per calibration size and percentile sampling are shown separately. The results are printed as a latex table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learner & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB & xGB &  \\\\\n",
      "CE & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce &  \\\\\n",
      "Cal.Size & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 &  \\\\\n",
      "Sample Size & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 &  \\\\\n",
      "\\hline\n",
      "colic & 0.12 & 0.13 & 0.15 & 0.16 & 0.24 & 0.13 & 0.14 & 0.15 & 0.17 & 0.24 & 0.13 & 0.14 & 0.16 & 0.18 & 0.25 & 0.15 & 0.17 & 0.18 & 0.21 & 0.32 & 0.15 & 0.17 & 0.19 & 0.22 & 0.31 & 0.16 & 0.17 & 0.20 & 0.22 & 0.34 & 60 \\\\\n",
      "creditA & 0.16 & 0.18 & 0.20 & 0.21 & 0.31 & 0.20 & 0.22 & 0.24 & 0.26 & 0.38 & 0.21 & 0.23 & 0.25 & 0.28 & 0.39 & 0.18 & 0.22 & 0.23 & 0.26 & 0.38 & 0.23 & 0.26 & 0.30 & 0.33 & 0.49 & 0.24 & 0.28 & 0.30 & 0.34 & 0.50 & 43 \\\\\n",
      "diabetes & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.02 & 0.03 & 0.04 & 0.06 & 0.11 & 0.02 & 0.04 & 0.05 & 0.06 & 0.12 & 0.02 & 0.03 & 0.04 & 0.06 & 0.11 & 9 \\\\\n",
      "german & 0.08 & 0.08 & 0.08 & 0.09 & 0.10 & 0.08 & 0.08 & 0.08 & 0.09 & 0.10 & 0.08 & 0.08 & 0.08 & 0.09 & 0.10 & 0.08 & 0.09 & 0.09 & 0.09 & 0.11 & 0.08 & 0.09 & 0.09 & 0.10 & 0.12 & 0.08 & 0.09 & 0.09 & 0.09 & 0.11 & 28 \\\\\n",
      "haberman & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 4 \\\\\n",
      "heartC & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.02 & 0.03 & 0.03 & 0.04 & 0.05 & 0.03 & 0.04 & 0.05 & 0.05 & 0.07 & 0.03 & 0.04 & 0.04 & 0.05 & 0.08 & 0.03 & 0.04 & 0.05 & 0.05 & 0.08 & 23 \\\\\n",
      "heartH & 0.02 & 0.02 & 0.02 & 0.02 & 0.04 & 0.02 & 0.02 & 0.03 & 0.03 & 0.04 & 0.02 & 0.02 & 0.03 & 0.03 & 0.04 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.03 & 0.03 & 0.04 & 0.04 & 0.06 & 0.03 & 0.03 & 0.04 & 0.04 & 0.06 & 21 \\\\\n",
      "heartS & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 14 \\\\\n",
      "hepati & 0.02 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.03 & 0.03 & 0.04 & 0.02 & 0.02 & 0.03 & 0.03 & 0.04 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 20 \\\\\n",
      "iono & 0.04 & 0.07 & 0.10 & 0.14 & 0.38 & 0.05 & 0.07 & 0.10 & 0.14 & 0.36 & 0.04 & 0.07 & 0.10 & 0.13 & 0.34 & 0.09 & 0.16 & 0.23 & 0.32 & 0.81 & 0.11 & 0.18 & 0.26 & 0.36 & 0.87 & 0.11 & 0.18 & 0.25 & 0.33 & 0.81 & 34 \\\\\n",
      "je4042 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.02 & 0.02 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.04 & 0.01 & 0.02 & 0.02 & 0.03 & 0.04 & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 9 \\\\\n",
      "je4243 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.03 & 0.05 & 0.01 & 0.02 & 0.02 & 0.03 & 0.05 & 0.02 & 0.02 & 0.03 & 0.03 & 0.06 & 9 \\\\\n",
      "kc1 & 0.05 & 0.09 & 0.13 & 0.18 & 0.48 & 0.05 & 0.09 & 0.14 & 0.18 & 0.49 & 0.05 & 0.09 & 0.13 & 0.18 & 0.49 & 0.13 & 0.25 & 0.37 & 0.49 & 1.19 & 0.13 & 0.25 & 0.36 & 0.48 & 1.15 & 0.13 & 0.25 & 0.37 & 0.49 & 1.15 & 22 \\\\\n",
      "kc2 & 0.02 & 0.04 & 0.07 & 0.07 & 0.16 & 0.03 & 0.04 & 0.05 & 0.06 & 0.16 & 0.03 & 0.04 & 0.06 & 0.07 & 0.17 & 0.05 & 0.08 & 0.11 & 0.12 & 0.28 & 0.05 & 0.08 & 0.11 & 0.15 & 0.32 & 0.07 & 0.10 & 0.13 & 0.18 & 0.40 & 22 \\\\\n",
      "kc3 & 0.06 & 0.10 & 0.15 & 0.20 & 0.57 & 0.06 & 0.10 & 0.16 & 0.21 & 0.56 & 0.06 & 0.10 & 0.16 & 0.21 & 0.56 & 0.13 & 0.23 & 0.33 & 0.44 & 1.08 & 0.12 & 0.21 & 0.31 & 0.41 & 1.00 & 0.12 & 0.22 & 0.31 & 0.42 & 1.01 & 40 \\\\\n",
      "liver & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 7 \\\\\n",
      "pc1req & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 9 \\\\\n",
      "pc4 & 0.17 & 0.34 & 0.49 & 0.69 & 1.83 & 0.18 & 0.32 & 0.49 & 0.67 & 1.78 & 0.19 & 0.34 & 0.51 & 0.71 & 1.94 & 0.38 & 0.71 & 1.05 & 1.45 & 3.53 & 0.40 & 0.73 & 1.09 & 1.47 & 3.53 & 0.47 & 0.85 & 1.25 & 1.68 & 3.97 & 38 \\\\\n",
      "sonar & 0.09 & 0.16 & 0.25 & 0.35 & 1.09 & 0.09 & 0.16 & 0.25 & 0.36 & 1.13 & 0.09 & 0.17 & 0.26 & 0.38 & 1.17 & 0.18 & 0.33 & 0.51 & 0.70 & 2.05 & 0.19 & 0.36 & 0.58 & 0.79 & 2.31 & 0.19 & 0.37 & 0.57 & 0.81 & 2.36 & 61 \\\\\n",
      "spect & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 23 \\\\\n",
      "spectf & 0.06 & 0.11 & 0.16 & 0.23 & 0.67 & 0.06 & 0.11 & 0.17 & 0.23 & 0.66 & 0.06 & 0.11 & 0.17 & 0.23 & 0.67 & 0.12 & 0.22 & 0.33 & 0.45 & 1.18 & 0.13 & 0.25 & 0.37 & 0.50 & 1.31 & 0.14 & 0.27 & 0.41 & 0.56 & 1.41 & 45 \\\\\n",
      "transfusion & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.02 & 0.02 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 5 \\\\\n",
      "ttt & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 28 \\\\\n",
      "vote & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 17 \\\\\n",
      "wbc & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 0.02 & 0.03 & 0.03 & 0.04 & 0.07 & 0.02 & 0.03 & 0.04 & 0.04 & 0.07 & 10 \\\\\n",
      "\\hline\n",
      "Average & 0.04 & 0.06 & 0.08 & 0.11 & 0.25 & 0.05 & 0.06 & 0.08 & 0.11 & 0.25 & 0.05 & 0.07 & 0.09 & 0.11 & 0.26 & 0.07 & 0.11 & 0.15 & 0.20 & 0.46 & 0.08 & 0.12 & 0.16 & 0.21 & 0.48 & 0.08 & 0.13 & 0.17 & 0.22 & 0.51 & 24.0 \\\\\n",
      "\n",
      "\n",
      "Learner & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF & RF &  \\\\\n",
      "CE & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & ce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce & cce &  \\\\\n",
      "Cal.Size & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.2 & 0.2 & 0.2 & 0.2 & 0.2 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 &  \\\\\n",
      "Sample Size & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 & 1 & 2 & 3 & 4 & 9 &  \\\\\n",
      "\\hline\n",
      "colic & 0.12 & 0.13 & 0.15 & 0.17 & 0.23 & 0.12 & 0.14 & 0.15 & 0.17 & 0.23 & 0.12 & 0.14 & 0.15 & 0.17 & 0.25 & 0.14 & 0.16 & 0.18 & 0.21 & 0.31 & 0.15 & 0.17 & 0.19 & 0.21 & 0.31 & 0.15 & 0.17 & 0.19 & 0.21 & 0.33 & 60 \\\\\n",
      "creditA & 0.16 & 0.18 & 0.19 & 0.22 & 0.31 & 0.19 & 0.22 & 0.24 & 0.27 & 0.38 & 0.20 & 0.23 & 0.25 & 0.28 & 0.39 & 0.19 & 0.21 & 0.23 & 0.25 & 0.39 & 0.23 & 0.26 & 0.29 & 0.32 & 0.49 & 0.24 & 0.27 & 0.30 & 0.33 & 0.50 & 43 \\\\\n",
      "diabetes & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.01 & 0.01 & 0.02 & 0.02 & 0.04 & 0.02 & 0.03 & 0.04 & 0.05 & 0.10 & 0.02 & 0.03 & 0.05 & 0.06 & 0.12 & 0.02 & 0.03 & 0.04 & 0.05 & 0.11 & 9 \\\\\n",
      "german & 0.08 & 0.08 & 0.08 & 0.09 & 0.10 & 0.08 & 0.08 & 0.08 & 0.08 & 0.10 & 0.08 & 0.08 & 0.08 & 0.09 & 0.10 & 0.08 & 0.08 & 0.09 & 0.09 & 0.10 & 0.08 & 0.09 & 0.09 & 0.09 & 0.11 & 0.08 & 0.09 & 0.09 & 0.09 & 0.11 & 28 \\\\\n",
      "haberman & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 4 \\\\\n",
      "heartC & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 0.02 & 0.02 & 0.03 & 0.03 & 0.05 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.03 & 0.03 & 0.04 & 0.04 & 0.07 & 0.03 & 0.03 & 0.04 & 0.05 & 0.07 & 0.03 & 0.04 & 0.04 & 0.05 & 0.07 & 23 \\\\\n",
      "heartH & 0.02 & 0.02 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.02 & 0.02 & 0.04 & 0.02 & 0.02 & 0.02 & 0.02 & 0.04 & 0.02 & 0.02 & 0.03 & 0.03 & 0.04 & 0.02 & 0.03 & 0.03 & 0.03 & 0.05 & 0.02 & 0.03 & 0.03 & 0.04 & 0.06 & 21 \\\\\n",
      "heartS & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.02 & 0.02 & 0.02 & 0.03 & 0.05 & 0.02 & 0.02 & 0.02 & 0.03 & 0.04 & 0.02 & 0.02 & 0.02 & 0.03 & 0.05 & 14 \\\\\n",
      "hepati & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.04 & 0.02 & 0.02 & 0.02 & 0.03 & 0.04 & 0.02 & 0.02 & 0.02 & 0.03 & 0.04 & 20 \\\\\n",
      "iono & 0.04 & 0.07 & 0.10 & 0.14 & 0.38 & 0.04 & 0.07 & 0.10 & 0.13 & 0.36 & 0.04 & 0.06 & 0.09 & 0.13 & 0.33 & 0.09 & 0.16 & 0.23 & 0.31 & 0.82 & 0.10 & 0.19 & 0.26 & 0.35 & 0.86 & 0.10 & 0.17 & 0.25 & 0.34 & 0.80 & 34 \\\\\n",
      "je4042 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.02 & 0.04 & 0.01 & 0.02 & 0.02 & 0.02 & 0.04 & 9 \\\\\n",
      "je4243 & 0.01 & 0.01 & 0.01 & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.03 & 0.01 & 0.02 & 0.02 & 0.03 & 0.05 & 0.01 & 0.02 & 0.02 & 0.02 & 0.04 & 0.01 & 0.02 & 0.02 & 0.03 & 0.05 & 9 \\\\\n",
      "kc1 & 0.05 & 0.09 & 0.13 & 0.18 & 0.48 & 0.05 & 0.09 & 0.13 & 0.18 & 0.49 & 0.05 & 0.09 & 0.13 & 0.18 & 0.49 & 0.13 & 0.25 & 0.37 & 0.50 & 1.19 & 0.13 & 0.25 & 0.37 & 0.48 & 1.16 & 0.13 & 0.25 & 0.36 & 0.49 & 1.15 & 22 \\\\\n",
      "kc2 & 0.02 & 0.03 & 0.05 & 0.06 & 0.15 & 0.02 & 0.03 & 0.05 & 0.06 & 0.15 & 0.03 & 0.04 & 0.05 & 0.06 & 0.17 & 0.04 & 0.06 & 0.09 & 0.11 & 0.26 & 0.05 & 0.07 & 0.10 & 0.13 & 0.32 & 0.06 & 0.10 & 0.13 & 0.17 & 0.39 & 22 \\\\\n",
      "kc3 & 0.05 & 0.10 & 0.14 & 0.20 & 0.56 & 0.05 & 0.10 & 0.15 & 0.21 & 0.57 & 0.06 & 0.10 & 0.15 & 0.20 & 0.54 & 0.12 & 0.22 & 0.33 & 0.43 & 1.08 & 0.11 & 0.21 & 0.30 & 0.39 & 1.00 & 0.12 & 0.21 & 0.31 & 0.41 & 1.02 & 40 \\\\\n",
      "liver & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 7 \\\\\n",
      "pc1req & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 9 \\\\\n",
      "pc4 & 0.17 & 0.32 & 0.48 & 0.67 & 1.82 & 0.18 & 0.32 & 0.48 & 0.66 & 1.76 & 0.19 & 0.35 & 0.51 & 0.71 & 1.91 & 0.38 & 0.71 & 1.04 & 1.41 & 3.45 & 0.40 & 0.73 & 1.09 & 1.46 & 3.51 & 0.50 & 0.86 & 1.25 & 1.68 & 3.95 & 38 \\\\\n",
      "sonar & 0.08 & 0.15 & 0.24 & 0.35 & 1.10 & 0.08 & 0.16 & 0.25 & 0.35 & 1.13 & 0.08 & 0.16 & 0.25 & 0.37 & 1.19 & 0.17 & 0.32 & 0.50 & 0.69 & 2.04 & 0.19 & 0.36 & 0.56 & 0.80 & 2.33 & 0.19 & 0.36 & 0.57 & 0.81 & 2.35 & 61 \\\\\n",
      "spect & 0.02 & 0.01 & 0.02 & 0.02 & 0.02 & 0.01 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 23 \\\\\n",
      "spectf & 0.06 & 0.11 & 0.16 & 0.22 & 0.67 & 0.06 & 0.10 & 0.16 & 0.22 & 0.66 & 0.06 & 0.11 & 0.16 & 0.22 & 0.67 & 0.12 & 0.21 & 0.33 & 0.44 & 1.18 & 0.13 & 0.24 & 0.36 & 0.50 & 1.30 & 0.14 & 0.27 & 0.40 & 0.55 & 1.41 & 45 \\\\\n",
      "transfusion & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.00 & 0.00 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.01 & 0.02 & 0.01 & 0.01 & 0.01 & 0.02 & 0.02 & 5 \\\\\n",
      "ttt & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 0.07 & 28 \\\\\n",
      "vote & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 0.02 & 17 \\\\\n",
      "wbc & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.03 & 0.01 & 0.01 & 0.02 & 0.02 & 0.03 & 0.01 & 0.02 & 0.02 & 0.03 & 0.05 & 0.02 & 0.02 & 0.03 & 0.04 & 0.06 & 0.02 & 0.03 & 0.03 & 0.04 & 0.06 & 10 \\\\\n",
      "\\hline\n",
      "Average & 0.04 & 0.06 & 0.08 & 0.10 & 0.25 & 0.04 & 0.06 & 0.08 & 0.10 & 0.25 & 0.04 & 0.06 & 0.08 & 0.11 & 0.26 & 0.07 & 0.11 & 0.15 & 0.19 & 0.46 & 0.07 & 0.12 & 0.16 & 0.21 & 0.48 & 0.08 & 0.12 & 0.17 & 0.22 & 0.51 & 24.0 \\\\\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "timer = []\n",
    "n = results['test_size']\n",
    "cal_sizes = results['calibration_sizes']\n",
    "perc_samples = results['sample_percentiles']\n",
    "average_time = {}\n",
    "average_time['num_features'] = []\n",
    "for a in ['xGB', 'RF']:\n",
    "    print('Learner & ', end='')\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for cal in ablation[key]:\n",
    "            for p in perc_samples:\n",
    "                average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+str(p)] = []\n",
    "                average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+str(p)] = []\n",
    "                print(f'{a} & ',end='')\n",
    "    print(' \\\\\\\\')\n",
    "    print('CE & ', end='')\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for cal in ablation[key]:\n",
    "            for p in perc_samples:\n",
    "                print(f'{key} & ',end='')\n",
    "    print(' \\\\\\\\')\n",
    "    print('Cal.Size & ', end='')\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for cal in ablation[key]:\n",
    "            for p in perc_samples:\n",
    "                print(f'{str(cal)} & ',end='')\n",
    "    print(' \\\\\\\\')\n",
    "    print('Sample Size & ', end='')\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for cal in ablation[key]:\n",
    "            for p in perc_samples:\n",
    "                print(f'{str(len(p))} & ',end='')\n",
    "    print(' \\\\\\\\\\n\\\\hline')\n",
    "\n",
    "    for d in np.sort(np.sort([k for k in results.keys()])):\n",
    "        if d in ['test_size', 'calibration_sizes', 'sample_percentiles']:\n",
    "            continue\n",
    "        print(d, end='')\n",
    "    # for a in results[d]:\n",
    "        n = len(results[d][a]['ablation']['ce'][cal_sizes[0]][str(perc_samples[0])][0])\n",
    "        for key in ['ce', 'cce']:  \n",
    "            a_time = results[d][a]['timer']\n",
    "            for cal in ablation[key]:\n",
    "                for p in ablation[key][cal]:          \n",
    "                    print(' & ', end='')\n",
    "                    res = np.mean([t/n for t in a_time[key][cal][p]])\n",
    "                    average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+p].append(res)\n",
    "                    average_time['num_features'].append(data_characteristics[d])\n",
    "                    print(f'{res:.2f}',end='')\n",
    "        print(f' & {data_characteristics[d]}', end='')\n",
    "        print(' \\\\\\\\')\n",
    "    print('\\\\hline\\nAverage', end='')\n",
    "\n",
    "    for key in ['ce', 'cce']: \n",
    "            for cal in ablation[key]:\n",
    "                for p in ablation[key][cal]:           \n",
    "                    print(' & ', end='')\n",
    "                    print(f'{np.mean(average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+p]):.2f}',end='')\n",
    "    print(f' & {np.mean(average_time[\"num_features\"]):.1f}', end='')\n",
    "    print(' \\\\\\\\\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Runtime Calibrations Sizes & xGB_ce_0.1 & xGB_ce_0.2 & xGB_ce_0.4 & xGB_cce_0.1 & xGB_cce_0.2 & xGB_cce_0.4 & RF_ce_0.1 & RF_ce_0.2 & RF_ce_0.4 & RF_cce_0.1 & RF_cce_0.2 & RF_cce_0.4 &  \\\\\n",
      "\\hline\n",
      "Average & 0.11 & 0.11 & 0.11 & 0.20 & 0.21 & 0.22 & 0.11 & 0.11 & 0.11 & 0.20 & 0.21 & 0.22 \\\\\n",
      "\n",
      "Runtime Sample Sizes & xGB_ce_1 & xGB_ce_2 & xGB_ce_3 & xGB_ce_4 & xGB_ce_9 & xGB_cce_1 & xGB_cce_2 & xGB_cce_3 & xGB_cce_4 & xGB_cce_9 & RF_ce_1 & RF_ce_2 & RF_ce_3 & RF_ce_4 & RF_ce_9 & RF_cce_1 & RF_cce_2 & RF_cce_3 & RF_cce_4 & RF_cce_9 &  \\\\\n",
      "\\hline\n",
      "Average & 0.05 & 0.06 & 0.09 & 0.11 & 0.25 & 0.08 & 0.12 & 0.16 & 0.21 & 0.49 & 0.04 & 0.06 & 0.08 & 0.10 & 0.25 & 0.07 & 0.12 & 0.16 & 0.21 & 0.48 \\\\\n"
     ]
    }
   ],
   "source": [
    "print('\\nRuntime Calibrations Sizes & ', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for cal in ablation[key]:\n",
    "            print(f'{a+\"_\"+key+\"_\"+str(cal)} & ',end='')\n",
    "print(' \\\\\\\\\\n\\\\hline\\nAverage', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']: \n",
    "        for cal in ablation[key]:      \n",
    "            print(' & ', end='')\n",
    "            print(f'{np.mean([average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+p] for p in ablation[key][cal]]):.2f}',end='')\n",
    "print(' \\\\\\\\')\n",
    "\n",
    "print('\\nRuntime Sample Sizes & ', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']:            \n",
    "        for p in perc_samples:\n",
    "            print(f'{a+\"_\"+key+\"_\"+str(len(p))} & ',end='')\n",
    "print(' \\\\\\\\\\n\\\\hline\\nAverage', end='')\n",
    "for a in ['xGB', 'RF']:\n",
    "    for key in ['ce', 'cce']:           \n",
    "        for p in perc_samples:     \n",
    "            print(' & ', end='')\n",
    "            print(f'{np.mean([average_time[a+\"_\"+key+\"_\"+str(cal)+\"_\"+str(p)] for cal in ablation[key]]):.2f}',end='')\n",
    "print(' \\\\\\\\')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results regarding runtime are as can be expected and the observations are summarized below:\n",
    "* The runtime increases almost linearly with the number of percentiles sampled for numerical features.\n",
    "* The runtime increases with the calibration size, even if the difference in runtime is fairly small.\n",
    "* CE is faster than CCE, as expected. The reason is that CCE will generally require additional calculations than CE, at least for numerical features.\n",
    "* The runtime tend to increase with the number of features, even if it is not a linear increase. This is due to the fact that categorical features with many categories are more expensive to compute, at least as long as the sampling size is small. Consequently, the number of categorical features together with the number of categories per feature is more important than the total number of features, especially when the sampling size (only affecting numerical features) is small.\n",
    "* A great part of the difference in runtime can be attributed to the underlying model, indicating that the choice of model used will have a great impact on runtime.\n",
    "### 5 Conclusion\n",
    "The individual algorithmic parameter that influence runtime most is the number of percentiles sampled for numerical features. As this tend to have a fairly small impact on the feature importance, it may be a reason to consider decreasing the number of percentiles sampled by default for numerical features (currently the default is 3: [25, 50, 75]). Using only the median ([50]) would on average reduce the runtime with almost half.\n",
    "#### Final note\n",
    "The core algorithm was updated early August 2024. The results reported here are using the code committed [chore: redirected __call__() to explain()](https://github.com/Moffran/calibrated_explanations/commit/068091186b6c383eafdae538b6b02defc78857b9). Average speedups range from 2-9 depending on setup, with substantially higher speedups of up to almost 40 times faster for individual datasets and setups. Previous results shown below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Runtime & xGB_ce_0.1_1 & xGB_ce_0.1_2 & xGB_ce_0.1_3 & xGB_ce_0.1_4 & xGB_ce_0.1_9 & xGB_ce_0.2_1 & xGB_ce_0.2_2 & xGB_ce_0.2_3 & xGB_ce_0.2_4 & xGB_ce_0.2_9 & xGB_ce_0.4_1 & xGB_ce_0.4_2 & xGB_ce_0.4_3 & xGB_ce_0.4_4 & xGB_ce_0.4_9 & xGB_cce_0.1_1 & xGB_cce_0.1_2 & xGB_cce_0.1_3 & xGB_cce_0.1_4 & xGB_cce_0.1_9 & xGB_cce_0.2_1 & xGB_cce_0.2_2 & xGB_cce_0.2_3 & xGB_cce_0.2_4 & xGB_cce_0.2_9 & xGB_cce_0.4_1 & xGB_cce_0.4_2 & xGB_cce_0.4_3 & xGB_cce_0.4_4 & xGB_cce_0.4_9 & RF_ce_0.1_1 & RF_ce_0.1_2 & RF_ce_0.1_3 & RF_ce_0.1_4 & RF_ce_0.1_9 & RF_ce_0.2_1 & RF_ce_0.2_2 & RF_ce_0.2_3 & RF_ce_0.2_4 & RF_ce_0.2_9 & RF_ce_0.4_1 & RF_ce_0.4_2 & RF_ce_0.4_3 & RF_ce_0.4_4 & RF_ce_0.4_9 & RF_cce_0.1_1 & RF_cce_0.1_2 & RF_cce_0.1_3 & RF_cce_0.1_4 & RF_cce_0.1_9 & RF_cce_0.2_1 & RF_cce_0.2_2 & RF_cce_0.2_3 & RF_cce_0.2_4 & RF_cce_0.2_9 & RF_cce_0.4_1 & RF_cce_0.4_2 & RF_cce_0.4_3 & RF_cce_0.4_4 & RF_cce_0.4_9 &  \\\\\n",
    "\\hline\n",
    "colic & 0.32 & 0.38 & 0.40 & 0.44 & 0.61 & 0.35 & 0.38 & 0.43 & 0.47 & 0.66 & 0.36 & 0.39 & 0.43 & 0.47 & 0.66 & 0.35 & 0.39 & 0.45 & 0.50 & 0.74 & 0.37 & 0.42 & 0.47 & 0.52 & 0.76 & 0.38 & 0.42 & 0.48 & 0.52 & 0.77 & 0.81 & 0.89 & 0.96 & 1.03 & 1.53 & 0.79 & 0.88 & 0.97 & 1.06 & 1.49 & 0.79 & 0.87 & 0.96 & 1.06 & 1.48 & 0.84 & 0.99 & 1.05 & 1.24 & 1.72 & 0.82 & 0.95 & 1.06 & 1.17 & 1.72 & 0.82 & 0.94 & 1.08 & 1.18 & 1.76 & 60 \\\\\n",
    "creditA & 0.26 & 0.29 & 0.33 & 0.36 & 0.51 & 0.30 & 0.34 & 0.37 & 0.40 & 0.56 & 0.32 & 0.36 & 0.39 & 0.42 & 0.56 & 0.28 & 0.31 & 0.34 & 0.38 & 0.55 & 0.31 & 0.36 & 0.40 & 0.44 & 0.64 & 0.34 & 0.38 & 0.42 & 0.45 & 0.64 & 0.64 & 0.79 & 0.86 & 0.94 & 1.31 & 0.79 & 0.86 & 0.96 & 1.04 & 1.43 & 0.86 & 0.94 & 0.99 & 1.08 & 1.47 & 0.73 & 0.81 & 0.94 & 1.00 & 1.47 & 0.83 & 0.91 & 1.02 & 1.13 & 1.66 & 0.90 & 0.98 & 1.08 & 1.21 & 1.70 & 43 \\\\\n",
    "diabetes & 0.05 & 0.10 & 0.15 & 0.19 & 0.42 & 0.06 & 0.10 & 0.15 & 0.19 & 0.43 & 0.06 & 0.11 & 0.16 & 0.20 & 0.43 & 0.07 & 0.13 & 0.18 & 0.24 & 0.51 & 0.08 & 0.14 & 0.20 & 0.26 & 0.61 & 0.08 & 0.15 & 0.21 & 0.26 & 0.56 & 0.13 & 0.27 & 0.36 & 0.48 & 1.04 & 0.13 & 0.25 & 0.36 & 0.50 & 1.05 & 0.13 & 0.24 & 0.36 & 0.47 & 1.03 & 0.20 & 0.32 & 0.47 & 0.62 & 1.29 & 0.18 & 0.33 & 0.48 & 0.67 & 1.35 & 0.17 & 0.32 & 0.47 & 0.60 & 1.35 & 9 \\\\\n",
    "german & 0.16 & 0.17 & 0.18 & 0.18 & 0.20 & 0.16 & 0.17 & 0.18 & 0.18 & 0.21 & 0.17 & 0.17 & 0.19 & 0.19 & 0.21 & 0.16 & 0.17 & 0.18 & 0.18 & 0.20 & 0.16 & 0.17 & 0.17 & 0.18 & 0.21 & 0.17 & 0.18 & 0.19 & 0.19 & 0.22 & 0.42 & 0.45 & 0.46 & 0.50 & 0.54 & 0.41 & 0.43 & 0.45 & 0.47 & 0.54 & 0.42 & 0.44 & 0.44 & 0.47 & 0.59 & 0.43 & 0.46 & 0.47 & 0.49 & 0.53 & 0.42 & 0.43 & 0.44 & 0.45 & 0.53 & 0.43 & 0.43 & 0.44 & 0.50 & 0.60 & 28 \\\\\n",
    "haberman & 0.02 & 0.04 & 0.05 & 0.07 & 0.15 & 0.02 & 0.04 & 0.05 & 0.07 & 0.15 & 0.02 & 0.04 & 0.06 & 0.07 & 0.15 & 0.02 & 0.04 & 0.05 & 0.07 & 0.15 & 0.02 & 0.04 & 0.06 & 0.07 & 0.15 & 0.02 & 0.04 & 0.06 & 0.07 & 0.15 & 0.05 & 0.09 & 0.13 & 0.18 & 0.39 & 0.05 & 0.09 & 0.14 & 0.18 & 0.38 & 0.05 & 0.09 & 0.13 & 0.18 & 0.38 & 0.05 & 0.09 & 0.13 & 0.17 & 0.39 & 0.05 & 0.09 & 0.13 & 0.18 & 0.38 & 0.05 & 0.09 & 0.13 & 0.17 & 0.38 & 4 \\\\\n",
    "heartC & 0.12 & 0.15 & 0.18 & 0.21 & 0.34 & 0.14 & 0.16 & 0.19 & 0.22 & 0.35 & 0.14 & 0.17 & 0.20 & 0.22 & 0.36 & 0.14 & 0.17 & 0.20 & 0.24 & 0.41 & 0.15 & 0.18 & 0.22 & 0.26 & 0.44 & 0.16 & 0.19 & 0.23 & 0.27 & 0.46 & 0.32 & 0.39 & 0.47 & 0.53 & 0.85 & 0.35 & 0.40 & 0.47 & 0.56 & 0.86 & 0.34 & 0.41 & 0.47 & 0.55 & 0.89 & 0.34 & 0.44 & 0.53 & 0.61 & 1.08 & 0.37 & 0.45 & 0.54 & 0.62 & 1.08 & 0.38 & 0.47 & 0.56 & 0.68 & 1.12 & 23 \\\\\n",
    "heartH & 0.10 & 0.13 & 0.15 & 0.18 & 0.30 & 0.12 & 0.15 & 0.17 & 0.20 & 0.31 & 0.12 & 0.15 & 0.17 & 0.20 & 0.32 & 0.11 & 0.14 & 0.18 & 0.20 & 0.36 & 0.13 & 0.16 & 0.20 & 0.23 & 0.40 & 0.13 & 0.17 & 0.20 & 0.24 & 0.43 & 0.27 & 0.33 & 0.40 & 0.45 & 0.77 & 0.29 & 0.36 & 0.42 & 0.49 & 0.79 & 0.29 & 0.36 & 0.43 & 0.47 & 0.79 & 0.29 & 0.37 & 0.45 & 0.53 & 0.90 & 0.32 & 0.40 & 0.50 & 0.58 & 1.01 & 0.31 & 0.42 & 0.49 & 0.58 & 1.04 & 21 \\\\\n",
    "heartS & 0.09 & 0.12 & 0.15 & 0.18 & 0.31 & 0.10 & 0.13 & 0.15 & 0.18 & 0.32 & 0.10 & 0.13 & 0.16 & 0.18 & 0.32 & 0.10 & 0.13 & 0.18 & 0.21 & 0.40 & 0.11 & 0.15 & 0.18 & 0.22 & 0.39 & 0.11 & 0.15 & 0.19 & 0.23 & 0.42 & 0.23 & 0.30 & 0.37 & 0.44 & 0.77 & 0.24 & 0.31 & 0.38 & 0.45 & 0.77 & 0.24 & 0.31 & 0.37 & 0.44 & 0.78 & 0.25 & 0.35 & 0.44 & 0.53 & 0.98 & 0.26 & 0.35 & 0.45 & 0.53 & 0.95 & 0.27 & 0.36 & 0.45 & 0.54 & 1.00 & 14 \\\\\n",
    "hepati & 0.10 & 0.13 & 0.16 & 0.19 & 0.35 & 0.12 & 0.15 & 0.18 & 0.21 & 0.36 & 0.12 & 0.15 & 0.19 & 0.22 & 0.38 & 0.11 & 0.15 & 0.18 & 0.24 & 0.42 & 0.13 & 0.17 & 0.21 & 0.25 & 0.44 & 0.13 & 0.18 & 0.22 & 0.26 & 0.47 & 0.26 & 0.35 & 0.41 & 0.53 & 0.92 & 0.30 & 0.37 & 0.44 & 0.49 & 0.89 & 0.28 & 0.35 & 0.44 & 0.53 & 0.93 & 0.28 & 0.38 & 0.48 & 0.60 & 1.09 & 0.32 & 0.43 & 0.51 & 0.59 & 1.07 & 0.30 & 0.42 & 0.52 & 0.63 & 1.14 & 20 \\\\\n",
    "iono & 0.22 & 0.39 & 0.57 & 0.72 & 1.54 & 0.22 & 0.41 & 0.62 & 0.75 & 1.52 & 0.23 & 0.38 & 0.53 & 0.68 & 1.40 & 0.30 & 0.51 & 0.74 & 0.96 & 2.07 & 0.29 & 0.52 & 0.82 & 1.10 & 2.13 & 0.31 & 0.54 & 0.76 & 1.02 & 2.06 & 0.50 & 0.96 & 1.44 & 1.82 & 3.77 & 0.49 & 0.92 & 1.29 & 1.78 & 3.91 & 0.50 & 0.88 & 1.32 & 1.63 & 3.46 & 0.64 & 1.51 & 1.85 & 2.36 & 5.15 & 0.64 & 1.23 & 1.84 & 2.49 & 5.40 & 0.67 & 1.25 & 1.85 & 2.42 & 5.16 & 34 \\\\\n",
    "je4042 & 0.06 & 0.10 & 0.13 & 0.17 & 0.35 & 0.06 & 0.10 & 0.13 & 0.17 & 0.35 & 0.07 & 0.11 & 0.15 & 0.20 & 0.38 & 0.06 & 0.10 & 0.13 & 0.17 & 0.35 & 0.06 & 0.10 & 0.13 & 0.17 & 0.36 & 0.07 & 0.12 & 0.15 & 0.20 & 0.42 & 0.17 & 0.27 & 0.37 & 0.46 & 0.96 & 0.16 & 0.29 & 0.39 & 0.50 & 1.01 & 0.18 & 0.29 & 0.40 & 0.52 & 1.02 & 0.17 & 0.27 & 0.36 & 0.47 & 1.00 & 0.16 & 0.28 & 0.38 & 0.48 & 1.05 & 0.18 & 0.30 & 0.40 & 0.51 & 1.04 & 9 \\\\\n",
    "je4243 & 0.06 & 0.09 & 0.13 & 0.17 & 0.35 & 0.06 & 0.10 & 0.13 & 0.17 & 0.37 & 0.07 & 0.10 & 0.15 & 0.18 & 0.38 & 0.06 & 0.10 & 0.14 & 0.17 & 0.36 & 0.06 & 0.10 & 0.14 & 0.17 & 0.35 & 0.07 & 0.11 & 0.15 & 0.18 & 0.38 & 0.15 & 0.25 & 0.34 & 0.43 & 0.96 & 0.15 & 0.26 & 0.35 & 0.46 & 0.94 & 0.16 & 0.26 & 0.36 & 0.45 & 0.94 & 0.15 & 0.24 & 0.34 & 0.43 & 0.96 & 0.15 & 0.26 & 0.35 & 0.45 & 0.96 & 0.16 & 0.26 & 0.35 & 0.44 & 0.94 & 9 \\\\\n",
    "kc1 & 0.14 & 0.26 & 0.38 & 0.51 & 1.07 & 0.15 & 0.28 & 0.37 & 0.50 & 1.06 & 0.14 & 0.26 & 0.36 & 0.48 & 1.04 & 0.18 & 0.34 & 0.48 & 0.63 & 1.34 & 0.18 & 0.33 & 0.46 & 0.61 & 1.28 & 0.17 & 0.31 & 0.45 & 0.58 & 1.29 & 0.32 & 0.60 & 0.91 & 1.18 & 2.57 & 0.31 & 0.59 & 0.89 & 1.17 & 2.50 & 0.32 & 0.59 & 0.87 & 1.16 & 2.69 & 0.41 & 0.79 & 1.14 & 1.49 & 3.22 & 0.40 & 0.75 & 1.09 & 1.45 & 3.08 & 0.38 & 0.72 & 1.07 & 1.33 & 3.12 & 22 \\\\\n",
    "kc2 & 0.12 & 0.23 & 0.34 & 0.45 & 0.96 & 0.13 & 0.25 & 0.35 & 0.45 & 0.99 & 0.15 & 0.26 & 0.37 & 0.47 & 1.00 & 0.15 & 0.29 & 0.41 & 0.55 & 1.18 & 0.17 & 0.30 & 0.44 & 0.58 & 1.22 & 0.19 & 0.34 & 0.49 & 0.63 & 1.30 & 0.30 & 0.57 & 0.83 & 1.09 & 2.46 & 0.33 & 0.58 & 0.88 & 1.14 & 2.44 & 0.33 & 0.59 & 0.86 & 1.12 & 2.43 & 0.36 & 0.70 & 1.02 & 1.33 & 3.42 & 0.39 & 0.73 & 1.13 & 1.44 & 2.97 & 0.42 & 0.78 & 1.12 & 1.48 & 3.29 & 22 \\\\\n",
    "kc3 & 0.24 & 0.46 & 0.66 & 0.86 & 1.85 & 0.25 & 0.46 & 0.66 & 0.87 & 1.82 & 0.26 & 0.46 & 0.67 & 0.85 & 1.83 & 0.30 & 0.56 & 0.81 & 1.04 & 2.22 & 0.30 & 0.55 & 0.79 & 1.00 & 2.13 & 0.31 & 0.57 & 0.81 & 1.04 & 2.18 & 0.58 & 1.10 & 1.53 & 2.09 & 4.48 & 0.60 & 1.09 & 1.56 & 2.02 & 4.41 & 0.59 & 1.11 & 1.52 & 1.98 & 4.25 & 0.70 & 1.31 & 1.96 & 2.50 & 5.57 & 0.68 & 1.28 & 1.81 & 2.37 & 5.13 & 0.72 & 1.29 & 1.82 & 2.38 & 5.11 & 40 \\\\\n",
    "liver & 0.04 & 0.07 & 0.10 & 0.13 & 0.28 & 0.04 & 0.07 & 0.10 & 0.13 & 0.26 & 0.04 & 0.07 & 0.10 & 0.14 & 0.28 & 0.05 & 0.09 & 0.13 & 0.17 & 0.37 & 0.05 & 0.09 & 0.13 & 0.16 & 0.34 & 0.05 & 0.10 & 0.14 & 0.19 & 0.39 & 0.10 & 0.18 & 0.26 & 0.34 & 0.74 & 0.09 & 0.17 & 0.25 & 0.33 & 0.66 & 0.09 & 0.18 & 0.26 & 0.35 & 0.72 & 0.12 & 0.24 & 0.34 & 0.44 & 0.98 & 0.11 & 0.22 & 0.32 & 0.41 & 0.89 & 0.14 & 0.24 & 0.37 & 0.48 & 1.04 & 7 \\\\\n",
    "pc1req & 0.07 & 0.07 & 0.07 & 0.08 & 0.10 & 0.07 & 0.07 & 0.08 & 0.09 & 0.11 & 0.08 & 0.09 & 0.10 & 0.10 & 0.13 & 0.06 & 0.07 & 0.07 & 0.07 & 0.10 & 0.07 & 0.08 & 0.08 & 0.09 & 0.11 & 0.08 & 0.08 & 0.10 & 0.10 & 0.14 & 0.16 & 0.18 & 0.19 & 0.20 & 0.27 & 0.18 & 0.19 & 0.21 & 0.22 & 0.30 & 0.20 & 0.21 & 0.23 & 0.24 & 0.31 & 0.17 & 0.17 & 0.19 & 0.20 & 0.26 & 0.18 & 0.19 & 0.22 & 0.23 & 0.30 & 0.20 & 0.21 & 0.23 & 0.25 & 0.31 & 9 \\\\\n",
    "pc4 & 0.22 & 0.40 & 0.58 & 0.76 & 1.54 & 0.25 & 0.41 & 0.58 & 0.75 & 1.60 & 0.26 & 0.46 & 0.63 & 0.87 & 1.76 & 0.28 & 0.51 & 0.73 & 0.95 & 1.96 & 0.30 & 0.53 & 0.74 & 0.99 & 2.09 & 0.33 & 0.57 & 0.81 & 1.09 & 2.22 & 0.58 & 1.04 & 1.50 & 1.95 & 3.91 & 0.60 & 1.02 & 1.47 & 1.90 & 3.88 & 0.56 & 0.98 & 1.39 & 1.96 & 4.22 & 0.72 & 1.28 & 1.86 & 2.44 & 5.03 & 0.73 & 1.31 & 1.89 & 2.46 & 4.92 & 0.70 & 1.25 & 1.77 & 2.47 & 5.16 & 38 \\\\\n",
    "sonar & 0.40 & 0.71 & 1.04 & 1.33 & 2.91 & 0.38 & 0.71 & 1.03 & 1.35 & 3.01 & 0.39 & 0.74 & 1.07 & 1.42 & 3.08 & 0.52 & 0.92 & 1.35 & 1.74 & 3.79 & 0.51 & 0.94 & 1.38 & 1.81 & 4.05 & 0.53 & 0.99 & 1.45 & 1.84 & 4.12 & 0.85 & 1.64 & 2.41 & 3.17 & 7.06 & 0.87 & 1.71 & 2.52 & 3.24 & 7.32 & 0.93 & 2.03 & 2.61 & 3.39 & 7.49 & 1.09 & 2.12 & 3.13 & 4.17 & 9.47 & 1.17 & 2.42 & 3.27 & 4.36 & 10.48 & 1.28 & 2.41 & 3.36 & 4.56 & 10.70 & 61 \\\\\n",
    "spect & 0.12 & 0.13 & 0.13 & 0.13 & 0.13 & 0.14 & 0.13 & 0.14 & 0.14 & 0.13 & 0.13 & 0.14 & 0.14 & 0.13 & 0.13 & 0.12 & 0.12 & 0.13 & 0.13 & 0.14 & 0.13 & 0.14 & 0.14 & 0.13 & 0.13 & 0.14 & 0.13 & 0.13 & 0.14 & 0.13 & 0.31 & 0.32 & 0.32 & 0.32 & 0.32 & 0.31 & 0.33 & 0.31 & 0.31 & 0.31 & 0.31 & 0.31 & 0.32 & 0.33 & 0.32 & 0.33 & 0.34 & 0.33 & 0.31 & 0.31 & 0.31 & 0.31 & 0.31 & 0.31 & 0.31 & 0.30 & 0.32 & 0.31 & 0.32 & 0.32 & 23 \\\\\n",
    "spectf & 0.27 & 0.51 & 0.75 & 0.98 & 2.18 & 0.30 & 0.55 & 0.80 & 1.06 & 2.35 & 0.31 & 0.58 & 0.80 & 1.08 & 2.33 & 0.27 & 0.52 & 0.74 & 0.99 & 2.24 & 0.30 & 0.56 & 0.79 & 1.07 & 2.32 & 0.31 & 0.56 & 0.81 & 1.05 & 2.26 & 0.65 & 1.23 & 1.87 & 2.62 & 6.05 & 0.64 & 1.31 & 1.92 & 2.43 & 5.66 & 0.65 & 1.31 & 1.98 & 2.63 & 5.75 & 0.64 & 1.24 & 1.87 & 2.44 & 5.97 & 0.64 & 1.30 & 1.86 & 2.46 & 5.79 & 0.70 & 1.35 & 2.07 & 2.56 & 5.84 & 45 \\\\\n",
    "transfusion & 0.03 & 0.05 & 0.07 & 0.10 & 0.21 & 0.03 & 0.05 & 0.08 & 0.10 & 0.22 & 0.03 & 0.05 & 0.07 & 0.10 & 0.22 & 0.03 & 0.05 & 0.08 & 0.10 & 0.21 & 0.03 & 0.05 & 0.08 & 0.12 & 0.21 & 0.03 & 0.05 & 0.08 & 0.10 & 0.22 & 0.07 & 0.12 & 0.19 & 0.24 & 0.53 & 0.07 & 0.12 & 0.18 & 0.24 & 0.52 & 0.06 & 0.12 & 0.18 & 0.23 & 0.50 & 0.07 & 0.13 & 0.18 & 0.24 & 0.52 & 0.07 & 0.12 & 0.18 & 0.24 & 0.51 & 0.07 & 0.12 & 0.18 & 0.23 & 0.51 & 5 \\\\\n",
    "ttt & 0.15 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.17 & 0.16 & 0.17 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.16 & 0.17 & 0.16 & 0.17 & 0.42 & 0.43 & 0.43 & 0.43 & 0.40 & 0.39 & 0.39 & 0.39 & 0.40 & 0.39 & 0.39 & 0.39 & 0.39 & 0.39 & 0.39 & 0.42 & 0.44 & 0.42 & 0.40 & 0.40 & 0.39 & 0.39 & 0.39 & 0.40 & 0.40 & 0.39 & 0.40 & 0.39 & 0.39 & 0.39 & 28 \\\\\n",
    "vote & 0.09 & 0.09 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.09 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.10 & 0.23 & 0.23 & 0.23 & 0.23 & 0.24 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.24 & 0.24 & 0.23 & 0.23 & 0.23 & 0.23 & 0.23 & 0.22 & 0.22 & 0.23 & 0.23 & 0.23 & 17 \\\\\n",
    "wbc & 0.07 & 0.11 & 0.15 & 0.18 & 0.35 & 0.07 & 0.11 & 0.14 & 0.17 & 0.35 & 0.07 & 0.11 & 0.14 & 0.18 & 0.35 & 0.07 & 0.12 & 0.16 & 0.19 & 0.37 & 0.08 & 0.13 & 0.17 & 0.22 & 0.44 & 0.09 & 0.14 & 0.18 & 0.23 & 0.44 & 0.17 & 0.27 & 0.38 & 0.45 & 0.91 & 0.16 & 0.26 & 0.35 & 0.44 & 0.90 & 0.17 & 0.28 & 0.37 & 0.48 & 0.92 & 0.18 & 0.30 & 0.40 & 0.48 & 0.97 & 0.19 & 0.32 & 0.43 & 0.57 & 1.14 & 0.20 & 0.34 & 0.45 & 0.59 & 1.09 & 10 \\\\\n",
    "\\hline\n",
    "\n",
    "Summary:\n",
    "Runtime Calibrations Sizes & xGB_ce_0.1 & xGB_ce_0.2 & xGB_ce_0.4 & xGB_cce_0.1 & xGB_cce_0.2 & xGB_cce_0.4 & RF_ce_0.1 & RF_ce_0.2 & RF_ce_0.4 & RF_cce_0.1 & RF_cce_0.2 & RF_cce_0.4 &  \\\\\n",
    "\\hline\n",
    "Average & 0.34 & 0.35 & 0.36 & 0.40 & 0.41 & 0.43 & 0.84 & 0.85 & 0.86 & 1.00 & 1.01 & 1.03 \\\\\n",
    "\n",
    "Runtime Sample Sizes & xGB_ce_1 & xGB_ce_2 & xGB_ce_3 & xGB_ce_4 & xGB_ce_9 & xGB_cce_1 & xGB_cce_2 & xGB_cce_3 & xGB_cce_4 & xGB_cce_9 & RF_ce_1 & RF_ce_2 & RF_ce_3 & RF_ce_4 & RF_ce_9 & RF_cce_1 & RF_cce_2 & RF_cce_3 & RF_cce_4 & RF_cce_9 &  \\\\\n",
    "\\hline\n",
    "Average & 0.15 & 0.22 & 0.29 & 0.36 & 0.71 & 0.17 & 0.26 & 0.35 & 0.43 & 0.85 & 0.36 & 0.54 & 0.71 & 0.89 & 1.75 & 0.40 & 0.63 & 0.83 & 1.05 & 2.14 \\\\"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "168dd73c7a7b76a0355e35f33a90e68c167b1dbb1e524891be00dd5c7b8524eb"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}