This_Is_A_Disaster_nbk.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-cajbWzcidnl"
      },
      "outputs": [],
      "source": [
        "# The notebook was last run on 23rd August '22."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wx-zINWkvyc6"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ishandandekar/This_Is_A_Disaster/blob/main/This_Is_A_Disaster_nbk.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uZnnFSgT25u8"
      },
      "source": [
        "# This_Is_A_Disaster  \n",
        "👋 Hello and welcome to the notebook. In this notebook, I make three models and make a submission to the **`nlp-getting-started`** Kaggle compition, with each of the model. This project was part of the [Zero-to-Mastery Tensorflow Developer course](https://zerotomastery.io/courses/learn-tensorflow/). In the end I have summarised the results of the submissions I made to the competition. This is the original notebook which I used for the competition. I have removed the outputs of the code cells to make the notebook look cleaner."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Sp7MV1LRQ9Z"
      },
      "outputs": [],
      "source": [
        "# Check for GPU\n",
        "!nvidia-smi -L"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "90x6YZVN32yc"
      },
      "source": [
        "## Step 0: About the problem "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0S3KoD35gOI"
      },
      "source": [
        "To predict whether a tweet is regarding a real disaster or not. If it is then mark the tweet as `1` else `0`.   \n",
        "\n",
        "Files:\n",
        "- train.csv - the training set\n",
        "- test.csv - the test set\n",
        "- sample_submission.csv - a sample submission file in the correct format  \n",
        "\n",
        "Columns:\n",
        "- id - a unique identifier for each tweet\n",
        "- text - the text of the tweet\n",
        "- location - the location the tweet was sent from (may be blank)\n",
        "- keyword - a particular keyword from the tweet (may be blank)\n",
        "- target - in train.csv only, this denotes whether a tweet is about a real - disaster (1) or not (0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfev0gpn5iEo"
      },
      "source": [
        "## Step 1: Get the data  \n",
        "The data is given officially by the competition organisers on Kaggle.   \n",
        "- Use Kaggle's API to download the data.\n",
        "- Get some utility functions which will further help us.\n",
        "- Configure data files to read using Python.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2uE1lOWZGtKH"
      },
      "outputs": [],
      "source": [
        "# Getting the helper functions script\n",
        "!wget https://raw.githubusercontent.com/ishandandekar/This_Is_A_Disaster/main/helper_functions.py --no-verbose\n",
        "\n",
        "# Get the necessary functions\n",
        "from helper_functions import unzip_data, plot_loss_curves, gen_metrics_report, get_metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qK9jUgx95k_a"
      },
      "outputs": [],
      "source": [
        "# Install the kaggle library\n",
        "!pip install -q kaggle\n",
        "\n",
        "# Upload the Kaggle API keys\n",
        "from google.colab import files\n",
        "files.upload()\n",
        "\n",
        "!mkdir ~/.kaggle\n",
        "\n",
        "# Copy the json file to the folder\n",
        "!cp kaggle.json ~/.kaggle\n",
        "\n",
        "# Change permissions for json to work with the Kaggle API\n",
        "!chmod 600 ~/.kaggle/kaggle.json\n",
        "\n",
        "# Download the dataset\n",
        "!kaggle competitions download -c nlp-getting-started\n",
        "\n",
        "# Unzip data\n",
        "unzip_data(\"nlp-getting-started.zip\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_to5c7-Y76Ut"
      },
      "source": [
        "## Step 2: Know more about the data\n",
        "- Reading the data using `Pandas` library.\n",
        "- Get the statistics about the data.\n",
        "- Check if data is imbalanced.\n",
        "- Visualizing sample data.\n",
        "- Split training data into train and validation sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ExM9auri8L8M"
      },
      "outputs": [],
      "source": [
        "# Importing library\n",
        "import pandas as pd\n",
        "\n",
        "# Reading the files in\n",
        "train_df= pd.read_csv(\"data/train.csv\",index_col=[0])\n",
        "test_df= pd.read_csv(\"data/test.csv\",index_col=[0])\n",
        "\n",
        "# Getting the first 5 rows of the training data\n",
        "train_df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e0XXrrMt9XsV"
      },
      "outputs": [],
      "source": [
        "# Getting a sample from training data\n",
        "train_df.sample(frac=1,random_state=42).head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JhDEezg3-vwi"
      },
      "outputs": [],
      "source": [
        "# Check for label imbalance\n",
        "train_df.target.value_counts().plot(kind=\"bar\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5hI2ZE44-3jF"
      },
      "outputs": [],
      "source": [
        "# Check train and test data size\n",
        "len(train_df),len(test_df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TauABBlo_A6i"
      },
      "outputs": [],
      "source": [
        "# Visualize some random training examples\n",
        "import random\n",
        "random_index = random.randint(0,len(train_df)-5)\n",
        "for row in train_df[[\"text\",\"target\"]][random_index:random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target: {target}\",\"(real disaster)\" if target>0 else \"(not real disaster)\")\n",
        "  print(f\"Text:\\n{text}\")\n",
        "  print(\"---\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TR9BghwU_i_S"
      },
      "outputs": [],
      "source": [
        "# Importing necessary function(s)\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_sentences,val_sentences,train_labels,val_labels = train_test_split(train_df[\"text\"].to_numpy(),train_df[\"target\"].to_numpy(),test_size=0.1,random_state=42)\n",
        "\n",
        "# Check the lengths of the dataframes\n",
        "print(f\"Length of train set: {len(train_sentences)}\")\n",
        "print(f\"Length of validation set: {len(val_sentences)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSHgoVmUBuhO"
      },
      "source": [
        "## Step 3: Building the first model\n",
        "- Use transfer learning to get a pretrained text-vectorization and embedding layer.\n",
        "- *(Optional)* Add more layers after the pretrained layer\n",
        "- Compile the model.\n",
        "- *(Optional)* Setup `ModelCheckpoint` and `EarlyStopping` callbacks.\n",
        "- Plot loss curves.\n",
        "- Make predictions on validation set.\n",
        "- Make predictions on test set.\n",
        "- Submit predictions on competition.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mfPjt86VCnp5"
      },
      "outputs": [],
      "source": [
        "# Get the pretrained embedding layer and using it as a Keras layer\n",
        "# Using the Universal-Sentence-Encoder, another option is GloVe (pretty famous too)!\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow as tf\n",
        "\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",input_shape=[],dtype=tf.string,trainable=False,name=\"USE\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9lBzXYmUDXpm"
      },
      "outputs": [],
      "source": [
        "# Making the model using Sequential API\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "model_0 = tf.keras.Sequential(\n",
        "    [\n",
        "     sentence_encoder_layer,\n",
        "     layers.Dense(64,activation=\"relu\"),\n",
        "     layers.Dense(1,activation=\"sigmoid\",name=\"output_layer\")\n",
        "    ],name=\"model_0_USE\"\n",
        ")\n",
        "\n",
        "# Compile the model\n",
        "model_0.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"Adam\",\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get the summary\n",
        "model_0.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pjbKBjzmEE1K"
      },
      "outputs": [],
      "source": [
        "# Setting up variables\n",
        "EPOCHS = 5\n",
        "\n",
        "# Create ModelCheckpoint callback to save a model's progress during training\n",
        "!mkdir -p model_checkpoint/model_0\n",
        "checkpoint_path = \"model_checkpoint/model_0.h5\"\n",
        "mc_callback_0 = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,\n",
        "                                                      monitor=\"val_accuracy\",\n",
        "                                                      save_best_only=True,\n",
        "                                                      save_weights_only=True,\n",
        "                                                      verbose=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "84qRYAB_Dt_T"
      },
      "outputs": [],
      "source": [
        "# Fitting the model on the training set\n",
        "model_0_history = model_0.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=EPOCHS,\n",
        "                              validation_data=(val_sentences,val_labels),\n",
        "                              callbacks=[mc_callback_0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jVM2QV1FQXxR"
      },
      "outputs": [],
      "source": [
        "# Plot loss curves\n",
        "plot_loss_curves(model_0_history)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pq5BuUFoFxZH"
      },
      "outputs": [],
      "source": [
        "# Load-in best weights saved using checkpoint callback\n",
        "model_0.load_weights(checkpoint_path)\n",
        "\n",
        "# Evaluate on the validation data\n",
        "model_0_pred_probs = model_0.predict(val_sentences)\n",
        "model_0_preds = tf.squeeze(tf.round(model_0_pred_probs))\n",
        "get_metrics(val_labels,model_0_preds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hcqwmP-CrzXn"
      },
      "outputs": [],
      "source": [
        "# Checking sample submission to format data for submission\n",
        "sample = pd.read_csv('data/sample_submission.csv')\n",
        "sample.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZoL7q0ZDyT_X"
      },
      "outputs": [],
      "source": [
        "test_df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t_UrqvCwrJN7"
      },
      "outputs": [],
      "source": [
        "# Making predictions on test set\n",
        "test_id,test_sentences = test_df.index,test_df['text'].to_numpy()\n",
        "model_0_pred_probs_test = model_0.predict(test_sentences)\n",
        "model_0_preds_test = tf.squeeze(tf.round(model_0_pred_probs_test))\n",
        "submission_0 = pd.DataFrame({'id':test_id,'target':list(map(int,model_0_preds_test))})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oo4xVB0qsLEr"
      },
      "outputs": [],
      "source": [
        "# Submitting the csv to the competition\n",
        "!mkdir submissions\n",
        "submission_0.to_csv(\"submissions/submission_0.csv\",index=False)\n",
        "\n",
        "# Below line has been commented as I have already made a submission\n",
        "# !kaggle competitions submit -c nlp-getting-started -f submissions/submission_0.csv -m \"23/8 First submission using model_0\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZZJn_we0v7qp"
      },
      "outputs": [],
      "source": [
        "submission_0.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pa9q1HV7J6ZH"
      },
      "source": [
        "## Step 4: Building a better model\n",
        "- Get a pretrained embedding layer and set `trainable=True`. This trains the parameters within the embedding layer too.\n",
        "- (Optional) Add more layers to fine-tune.\n",
        "- Compile the model.\n",
        "- (Optional) Setup `ModelCheckpoint` and `EarlyStopping` callbacks.\n",
        "- Plot loss curves.\n",
        "- Make predictions on validation set.\n",
        "- Make predictions on test set.\n",
        "- Submit predictions to competition."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zTEGOr4kO8NB"
      },
      "outputs": [],
      "source": [
        "# Get the pretrained embedding layer and using it as a Keras layer\n",
        "# Setup trainable as True so the params within this layer can be changed while training\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",input_shape=[],dtype=tf.string,trainable=True,name=\"USE\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cSA7-pEWPHwt"
      },
      "outputs": [],
      "source": [
        "# Making the model using Sequential API\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "model_1 = tf.keras.Sequential(\n",
        "    [\n",
        "     sentence_encoder_layer,\n",
        "     layers.Dense(64,activation=\"relu\"),\n",
        "     layers.Dropout(0.2),\n",
        "     layers.Dense(64,activation=\"relu\"),\n",
        "     layers.Dense(1,activation=\"sigmoid\",name=\"output_layer\")\n",
        "    ],name=\"model_1_trainable\"\n",
        ")\n",
        "\n",
        "# Compile the model\n",
        "model_1.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"Adam\",\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get the model summary\n",
        "model_1.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HDQPcPmYPgSl"
      },
      "outputs": [],
      "source": [
        "# Setting up variables\n",
        "EPOCHS = 10\n",
        "\n",
        "# Create ModelCheckpoint callback to save a model's progress during training\n",
        "checkpoint_path = \"model_checkpoints/model_1_cp.ckpt\"\n",
        "mc_callback_1 = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,\n",
        "                                                   monitor=\"val_accuracy\",\n",
        "                                                   save_best_only=True,\n",
        "                                                   save_weights_only=True,\n",
        "                                                   verbose=0)\n",
        "\n",
        "es_callback = tf.keras.callbacks.EarlyStopping(\n",
        "    monitor=\"val_loss\",\n",
        "    patience=2,\n",
        "    verbose=0,\n",
        "    mode=\"auto\",\n",
        "    restore_best_weights=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7u2QtZolPxLF"
      },
      "outputs": [],
      "source": [
        "# Training the model\n",
        "model_1_history = model_1.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=EPOCHS,\n",
        "                              validation_data=(val_sentences,val_labels),\n",
        "                              callbacks=[mc_callback_1,es_callback])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9PSiwWqQP-Pk"
      },
      "outputs": [],
      "source": [
        "# Plot loss curves\n",
        "plot_loss_curves(model_1_history)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6svfbRpFQgyB"
      },
      "outputs": [],
      "source": [
        "# Load-in best weights saved using checkpoint callback\n",
        "model_1.load_weights(checkpoint_path)\n",
        "\n",
        "# Evaluate on the validation data\n",
        "model_1_pred_probs = model_1.predict(val_sentences)\n",
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_probs))\n",
        "gen_metrics_report(val_labels,model_1_preds)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8WahKQjZs17w"
      },
      "outputs": [],
      "source": [
        "# Making predictions on test set\n",
        "model_1_pred_probs_test = model_1.predict(test_sentences)\n",
        "model_1_preds_test = tf.squeeze(tf.round(model_1_pred_probs_test))\n",
        "submission_1 = pd.DataFrame({'id':test_id,'target':list(map(int,model_1_preds_test))})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bV4gB-UtwpGL"
      },
      "outputs": [],
      "source": [
        "# Making submission to the competition\n",
        "submission_0.to_csv(\"submissions/submission_1.csv\",index=False)\n",
        "\n",
        "# Below line has been commented, as the data has been already submitted\n",
        "# !kaggle competitions submit -c nlp-getting-started -f submissions/submission_1.csv -m \"23/8 Second submission using model_1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jlkJw2sqSdK"
      },
      "source": [
        "## Step 5: Building a *better* better model\n",
        "- Get the same model in **step 5**.\n",
        "- Add callbacks.\n",
        "- Train the model on whole training set (training set before split).\n",
        "- Make predictions on test set.\n",
        "- Submit predictions on competition."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5vQZcv9jyvhi"
      },
      "outputs": [],
      "source": [
        "# Cloning model_1\n",
        "model_2 = tf.keras.models.clone_model(model_1)\n",
        "\n",
        "# Compile the model\n",
        "model_2.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"Adam\",\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get the model summary\n",
        "model_2.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DAKTu29Xzd9g"
      },
      "outputs": [],
      "source": [
        "# Setting up variables\n",
        "EPOCHS = 10"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iNBjRYDIz0SO"
      },
      "outputs": [],
      "source": [
        "# Formatting train data\n",
        "train_sentences_full,train_labels_full = train_df[\"text\"],train_df[\"target\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2BsJYdPqziR4"
      },
      "outputs": [],
      "source": [
        "# Training the model\n",
        "model_2_history = model_2.fit(train_sentences_full,\n",
        "                              train_labels_full,\n",
        "                              epochs=EPOCHS)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KvLlC45y0mpl"
      },
      "outputs": [],
      "source": [
        "# Can't plot loss curve as the history object doesn't have val_loss and accuracy in it's dictionary\n",
        "# plot_loss_curves(model_2_history)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xr8ryLdd2CHh"
      },
      "outputs": [],
      "source": [
        "# Load-in best weights saved using checkpoint callback\n",
        "# model_2.load_weights(checkpoint_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O-2XxM437JlO"
      },
      "outputs": [],
      "source": [
        "# Making predictions on test set\n",
        "model_2_pred_probs_test = model_2.predict(test_sentences)\n",
        "model_2_preds_test = tf.squeeze(tf.round(model_2_pred_probs_test))\n",
        "submission_2 = pd.DataFrame({'id':test_id,'target':list(map(int,model_2_preds_test))})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_fFvnk06-v7F"
      },
      "outputs": [],
      "source": [
        "# Making submission to the competition\n",
        "submission_2.to_csv(\"submissions/submission_2.csv\",index=False)\n",
        "\n",
        "# Below line has been commented, as the data has been already submitted\n",
        "# !kaggle competitions submit -c nlp-getting-started -f submissions/submission_2.csv -m \"23/8 Third submission using model_2\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBbdjV4KBdEj"
      },
      "source": [
        "## Step 6: Comparing the two models' results\n",
        "- Check the results on Kaggle's website\n",
        "- Analyze results\n",
        "- How to improve these\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpT14lGbuN1f"
      },
      "source": [
        "1. **Model 0** : It had the simplest architecture amongest the other models. Various callbacks were used to make the training process better. Model was fit on 90% of the actual training data. Model got a score of **0.80570** on test data (taken from the Kaggle evaluation).  \n",
        "\n",
        "1. **Model 1** : It had a little different architecture than the previous model. Adding a Dropout layer and a Dense was thought to be advantageous so as to add more trainable parameters. But due to the change in this large number of trainable parameters the data model could've overfit. Model got a score of **0.80570** on test data (taken from the Kaggle evaluation). Model got the same score as the previous one. Adding more trainable parameters should've affected the score.  \n",
        "\n",
        "1. **Model 2** : The model had the same architecture as the previous model. The change in training this model, was to train on the full training set. This meant there was no validation set for this. Due to which, no callbacks could be added. The model while training was highly vulnerable to overfitting. Model got a score of **0.76984** on test data (taken from the Kaggle evaluation). The model has clearly overfit on the training set.\n",
        "\n",
        "Clearly the models could've been better. Change in encoders such as using GloVe intead could give a different result. Adding more layers and dropouts could change the results. Analyzing the data more and creating features could also help."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "90x6YZVN32yc",
        "_to5c7-Y76Ut",
        "hBbdjV4KBdEj"
      ],
      "name": "This_Is_A_Disaster_nbk.ipynb",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.9.7 ('foodvision')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.7"
    },
    "vscode": {
      "interpreter": {
        "hash": "d335e26c07bdd5b05df8c44a5d3f59ef2bdd9cdfad27507d4d274fb6f051ac29"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}