transformer/Transformer_Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformer: Attention is all you need\n",
    "\n",
    "This jupyter notebook is Tensorflow version implemented in the paper [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf). The task is translating a source human-readable datetime to a target fixed datetime format **yyyy-mm-dd**, e.g: \"24th Aug 19\" -> \"2019-08-24\". Best way to start implement a model from scratch is using small dataset and non-complex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tqdm\n",
    "from faker import Faker\n",
    "from babel.dates import format_date\n",
    "from nmt_utils import load_dataset_v2, preprocess_data, string_to_int, int_to_string, softmax\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn.model_selection import train_test_split\n",
    "import os\n",
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 40000/40000 [00:01<00:00, 32013.67it/s]\n"
     ]
    }
   ],
   "source": [
    "m = 40000\n",
    "dataset, human_vocab, machine_vocab, inv_machine_vocab = load_dataset_v2(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<pad>': 0,\n",
       " '<unk>': 1,\n",
       " ' ': 2,\n",
       " '.': 3,\n",
       " '/': 4,\n",
       " '0': 5,\n",
       " '1': 6,\n",
       " '2': 7,\n",
       " '3': 8,\n",
       " '4': 9,\n",
       " '5': 10,\n",
       " '6': 11,\n",
       " '7': 12,\n",
       " '8': 13,\n",
       " '9': 14,\n",
       " 'a': 15,\n",
       " 'b': 16,\n",
       " 'c': 17,\n",
       " 'd': 18,\n",
       " 'e': 19,\n",
       " 'f': 20,\n",
       " 'g': 21,\n",
       " 'h': 22,\n",
       " 'i': 23,\n",
       " 'j': 24,\n",
       " 'l': 25,\n",
       " 'm': 26,\n",
       " 'n': 27,\n",
       " 'o': 28,\n",
       " 'p': 29,\n",
       " 'r': 30,\n",
       " 's': 31,\n",
       " 't': 32,\n",
       " 'u': 33,\n",
       " 'v': 34,\n",
       " 'w': 35,\n",
       " 'y': 36}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'#': 0,\n",
       " '-': 1,\n",
       " '0': 2,\n",
       " '1': 3,\n",
       " '2': 4,\n",
       " '3': 5,\n",
       " '4': 6,\n",
       " '5': 7,\n",
       " '6': 8,\n",
       " '7': 9,\n",
       " '8': 10,\n",
       " '9': 11}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "machine_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape: (40000, 30)\n",
      "Y.shape: (40000, 11)\n"
     ]
    }
   ],
   "source": [
    "Tx = 30\n",
    "Ty = 10\n",
    "\n",
    "X, Y = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty+1)\n",
    "\n",
    "print(\"X.shape:\", X.shape)\n",
    "print(\"Y.shape:\", Y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer model with Tensorflow.\n",
    "\n",
    "### Hyperparameter:\n",
    "\n",
    "$d_{model}$: dimension of word embeding, output of **Multi-head Attention** layer, output of **Feed Forward** layer.\n",
    "\n",
    "$d_k$: dimension of matrix Q, K\n",
    "\n",
    "$d_v$: dimension of matrix V\n",
    "\n",
    "$d_{ff}$: dimension of intermediate **Feed forward** layer\n",
    "\n",
    "$h$: number of heads at each block.\n",
    "\n",
    "\n",
    "### Positional Encoding:\n",
    "\n",
    "Since the Transformer model isn't sequential model like RNN and CNN. The computation is parallel over all input sentence flow from Embedding Layer, so we need to compute the relative or absolute position between the words. The author use non-trainable/fixed signusoid function:\n",
    "\n",
    "$$PE_{(pos, 2i)} = sin\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right) \\mbox{this corresponding to the even indices}$$\n",
    "$$PE_{(pos, 2i+1)} = cos\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right) \\mbox{this corresponding to the odd indices}$$\n",
    "\n",
    "where $pos$ is position in the sequence and $i$ is the dimension.\n",
    "\n",
    "\n",
    "### Scaled Dot-Product Attention:\n",
    "\n",
    "<img style=\"width:300px; height:300px\" src=\"https://i.imgur.com/HuXNlr0.png\" />\n",
    "\n",
    "$$Attention(Q, K, V) = softmax\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$$\n",
    "\n",
    "### (Encoder-Decoder) Multi-Head Attention:\n",
    "\n",
    "<img style=\"weight:300px; height:300px\" src=\"https://i.imgur.com/vgfOLR2.png\" />\n",
    "\n",
    "$$MultiHead(Q, K, V) = Concat(head_1, head_2, ..., head_h)W^O$$\n",
    "$$\\mbox{where } head_i = Attention(Q, K, V)$$\n",
    "\n",
    "### Feed forward:\n",
    "\n",
    "$$FFN(x) = max(0, xW_1 + b_1)W_2 + b_2$$\n",
    "\n",
    "### Encoder blocks:\n",
    "\n",
    "Each encoder block include 2 layers: **Multi-head Attention Mechanism** and **Position-wise Feed Forward**, respestively. Output at each layer use residual connection with its input followed by [Layer Normalization](https://arxiv.org/pdf/1607.06450.pdf): $LayerNorm(x + f(x))$\n",
    "\n",
    "### Decoder blocks:\n",
    "\n",
    "Each decoder block includes 3 layers: **Multi-head Attention Mechanism**, **Encoder-Decoder Multi-head Attention** and **Position-wise Feed Forward**. Same as **Encoder** blocks, output at each layer use residual connection with its input follow by Layer Normalization.\n",
    "\n",
    "<img src=\"https://i.imgur.com/1NUHvLi.jpg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomLinear(nn.Linear):\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        nn.init.xavier_uniform_(self.weight)\n",
    "        if self.bias is not None:\n",
    "            nn.init.zeros_(self.bias)\n",
    "\n",
    "class Transformer(nn.Module):\n",
    "\n",
    "    def __init__(self, num_blocks, num_heads, vocab_size, seq_len, d_model, d_k, d_v, d_ff, device):\n",
    "        super(Transformer, self).__init__()\n",
    "        self.num_blocks = num_blocks\n",
    "        self.num_heads = num_heads\n",
    "        self.vocab_size = vocab_size\n",
    "        self.d_model = d_model\n",
    "        self.seq_len = seq_len\n",
    "        self.d_k = d_k\n",
    "        self.d_v = d_v\n",
    "        self.d_ff = d_ff\n",
    "        self.device = device\n",
    "        self.word_embed = nn.Embedding(num_embeddings=vocab_size, embedding_dim=d_model).to(device)\n",
    "\n",
    "    def _init_structure(self, decoder_part=False):\n",
    "        assert not hasattr(self, \"pos_enc\"), \"The structure is initialized already.\"\n",
    "        self.pos_embed = torch.zeros(size=(self.seq_len, self.d_model), requires_grad=False, device=self.device)\n",
    "        for pos in range(self.seq_len):\n",
    "            for i in range(0, self.d_model, 2):\n",
    "                self.pos_embed[pos, i] = torch.sin(torch.Tensor([pos / (10000 ** (i/self.d_model))]))\n",
    "                self.pos_embed[pos, i + 1] = torch.cos(torch.Tensor([pos / (10000 ** (i/self.d_model))]))\n",
    "        self.pos_embed = self.pos_embed.unsqueeze(0)\n",
    "\n",
    "        if decoder_part:\n",
    "            self.mask = [[0]*(i+1) + [-1e9]*(self.seq_len-(i+1)) for i in range(self.seq_len)]\n",
    "            self.mask = torch.tensor([self.mask], requires_grad=False).to(self.device)\n",
    "\n",
    "        for block_id in range(self.num_blocks):\n",
    "            # Self-attention sub-layer\n",
    "            setattr(self, \"Q\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_model, out_features=self.d_k * self.num_heads).to(self.device))\n",
    "            setattr(self, \"K\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_model, out_features=self.d_k * self.num_heads).to(self.device))\n",
    "            setattr(self, \"V\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_model, out_features=self.d_v * self.num_heads).to(self.device))\n",
    "                        \n",
    "            setattr(self, \"LN1\" + str(block_id), nn.LayerNorm(self.d_model).to(self.device))\n",
    "            # ---------------------------\n",
    "            \n",
    "            # Encoder-Decoder attention sub-layer\n",
    "            if decoder_part:\n",
    "                setattr(self, \"Qconn\" + str(block_id), \n",
    "                        CustomLinear(in_features=self.d_model, out_features=self.d_k * self.num_heads).to(self.device))\n",
    "                setattr(self, \"Kconn\" + str(block_id), \n",
    "                        CustomLinear(in_features=self.d_model, out_features=self.d_k * self.num_heads).to(self.device))\n",
    "                setattr(self, \"Vconn\" + str(block_id), \n",
    "                        CustomLinear(in_features=self.d_model, out_features=self.d_v * self.num_heads).to(self.device))\n",
    "                \n",
    "                setattr(self, \"LN2\" + str(block_id), nn.LayerNorm(self.d_v * self.num_heads).to(self.device))\n",
    "            # -----------------------------------\n",
    "            \n",
    "            # Layer multi-head attention output\n",
    "            setattr(self, \"O\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_v * self.num_heads, out_features=self.d_model).to(self.device))\n",
    "            # Layer FNN 1\n",
    "            setattr(self, \"FNN1\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_model, out_features=self.d_ff).to(self.device))\n",
    "            # Layer FNN 2\n",
    "            setattr(self, \"FNN2\" + str(block_id), \n",
    "                    CustomLinear(in_features=self.d_ff, out_features=self.d_model).to(self.device))\n",
    "            \n",
    "            setattr(self, \"LN3\" + str(block_id), nn.LayerNorm(self.d_model).to(self.device))\n",
    "\n",
    "    def _compute_multi_head_attention(self, Q, K, V, block_id, mask=False, connection_head=False):\n",
    "        if connection_head:\n",
    "            Q = getattr(self, \"Qconn\" + str(block_id))(Q)\n",
    "            K = getattr(self, \"Qconn\" + str(block_id))(K)\n",
    "            V = getattr(self, \"Qconn\" + str(block_id))(V)\n",
    "        else:\n",
    "            Q = getattr(self, \"Q\" + str(block_id))(Q)\n",
    "            K = getattr(self, \"Q\" + str(block_id))(K)\n",
    "            V = getattr(self, \"Q\" + str(block_id))(V)\n",
    "        QK = torch.einsum(\"ntk,nyk->nty\", Q, K)\n",
    "        if mask:\n",
    "            # apply mask to QK, prevent the affect of feature words to current word in decoder.\n",
    "            QK = QK + self.mask[:, :QK.shape[1], :QK.shape[2]]\n",
    "        QK = torch.softmax(QK/torch.sqrt(torch.Tensor([self.d_model]).to(self.device)), dim=-1)\n",
    "        atts = torch.einsum(\"nty,nyv->ntv\", QK, V)\n",
    "        O = getattr(self, \"O\" + str(block_id))(atts)\n",
    "        return O\n",
    "\n",
    "    def _compute_layer_norm(self, fX, X, name):\n",
    "        LN = getattr(self, name)(fX + X)\n",
    "        return LN\n",
    "\n",
    "    def _compute_fnn(self, X, block_id):\n",
    "        ffn1 = nn.functional.relu(getattr(self, \"FNN1\" + str(block_id))(X)) \n",
    "        ffn2 = getattr(self, \"FNN2\" + str(block_id))(ffn1)\n",
    "        return ffn2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(Transformer):\n",
    "\n",
    "    def __init__(self, num_blocks, num_heads, vocab_size, seq_len, d_model, d_k, d_v, d_ff, device):\n",
    "        super(Encoder, self).__init__(num_blocks, num_heads, vocab_size, seq_len, d_model, d_k, d_v, d_ff, device)\n",
    "        self._init_structure()\n",
    "\n",
    "    def forward(self, X):\n",
    "        x = self.word_embed(X)\n",
    "        x = x + self.pos_embed\n",
    "\n",
    "        for block_id in range(self.num_blocks):\n",
    "            fx = self._compute_multi_head_attention(x, x, x, block_id)\n",
    "            x = self._compute_layer_norm(fx, x, \"LN1\" + str(block_id))\n",
    " \n",
    "            fx = self._compute_fnn(x, block_id)\n",
    "            x = self._compute_layer_norm(fx, x, \"LN3\" + str(block_id))\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(Transformer):\n",
    "    \n",
    "    def __init__(self, num_blocks, num_heads, vocab_size, seq_len, d_model, d_k, d_v, d_ff, device):\n",
    "        super(Decoder, self).__init__(num_blocks, num_heads, vocab_size, seq_len, d_model, d_k, d_v, d_ff, device)\n",
    "        self._init_structure(decoder_part=True)\n",
    "        self.O_last = CustomLinear(in_features=d_model, out_features=vocab_size).to(self.device)\n",
    "\n",
    "    def forward(self, X, encoder_output):\n",
    "        x = self.word_embed(X)\n",
    "        x = x + self.pos_embed[:, :x.shape[1], :]\n",
    "\n",
    "        for block_id in range(self.num_blocks):\n",
    "            fx = self._compute_multi_head_attention(x, x, x, block_id, mask=True)\n",
    "            x = self._compute_layer_norm(fx, x, \"LN1\" + str(block_id))\n",
    "\n",
    "            fx = self._compute_multi_head_attention(x, encoder_output, encoder_output, block_id, connection_head=True)\n",
    "            x = self._compute_layer_norm(fx, x, \"LN2\" + str(block_id))\n",
    "\n",
    "            fx = self._compute_fnn(x, block_id)\n",
    "            x = self._compute_layer_norm(fx, x, \"LN3\" + str(block_id))\n",
    "        \n",
    "        logits = self.O_last(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(logits, target):\n",
    "    logits = logits.reshape(-1, logits.shape[-1])\n",
    "    target = target.reshape(-1)\n",
    "    return torch.nn.functional.cross_entropy(logits, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define hyperparameter for Transformer Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_BLOCKS = 2\n",
    "NUM_HEADS = 2\n",
    "DIMENSION_MODEL = 32\n",
    "DIMENSION_K = 16\n",
    "DIMENSION_V = 16\n",
    "DIMENSION_FF = 64\n",
    "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.Tensor(X).long().to(DEVICE)\n",
    "Y = torch.Tensor(Y).long().to(DEVICE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = Encoder(num_blocks=NUM_BLOCKS, num_heads=NUM_HEADS, vocab_size=len(human_vocab), seq_len=Tx, \n",
    "                  d_model=DIMENSION_MODEL, d_k=DIMENSION_K, d_v=DIMENSION_V, d_ff=DIMENSION_FF, device=DEVICE)\n",
    "\n",
    "decoder = Decoder(num_blocks=NUM_BLOCKS, num_heads=NUM_HEADS, vocab_size=len(machine_vocab), seq_len=Ty, \n",
    "                  d_model=DIMENSION_MODEL, d_k=DIMENSION_K, d_v=DIMENSION_V, d_ff=DIMENSION_FF, device=DEVICE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 2\n",
    "batch_size = 64\n",
    "num_batches = X.shape[0]//batch_size if X.shape[0] % batch_size == 0 else X.shape[0]//batch_size + 1\n",
    "data = torch.cat((X, Y), dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "params_update = list(encoder.parameters()) + list(decoder.parameters())\n",
    "optimizer = torch.optim.Adam(params_update, lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0d2acff39244acf82eb7b4143d9e342",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 1', max=625, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdbd4882a4cc45d599c1bff10af99b2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch 2', max=625, style=ProgressStyle(description_width='ini…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for e in range(epochs):\n",
    "    \n",
    "    data = data[torch.randperm(data.shape[0])]\n",
    "    \n",
    "    X, Y = data[:, :Tx], data[:, Tx:]\n",
    "    \n",
    "    pbar = tqdm.tqdm_notebook(range(0, num_batches), desc=\"Epoch \" + str(e+1))\n",
    "    \n",
    "    train_loss = 0\n",
    "    \n",
    "    for it in pbar:\n",
    "        loss = 0\n",
    "        start = it*batch_size\n",
    "        end = (it+1)*batch_size\n",
    "        \n",
    "        encoder_output = encoder(X[start:end])\n",
    "            \n",
    "        logits = decoder(Y[start:end, :-1], encoder_output)\n",
    "            \n",
    "        loss = loss_function(logits, Y[start:end, 1:])\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "            \n",
    "        train_loss += float(loss)\n",
    "        \n",
    "        pbar.set_description(\"Epoch %s - Training loss: %f\" % (e+1, (train_loss / (it+1))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "source: 3 May 1979\n",
      "parallel output: 1979-05-03\n",
      "-----------------------------------------------\n",
      "source: 5 April 09\n",
      "parallel output: 1990-04-05\n",
      "-----------------------------------------------\n",
      "source: 21th of August 2016\n",
      "parallel output: 2016-08-21\n",
      "-----------------------------------------------\n",
      "source: Tue 10 Jul 2007\n",
      "parallel output: 2007-07-10\n",
      "-----------------------------------------------\n",
      "source: Saturday May 9 2018\n",
      "parallel output: 2018-05-09\n",
      "-----------------------------------------------\n",
      "source: March 3 2001\n",
      "parallel output: 2001-03-03\n",
      "-----------------------------------------------\n",
      "source: March 3rd 2001\n",
      "parallel output: 2001-03-03\n",
      "-----------------------------------------------\n",
      "source: 1 March 2001\n",
      "parallel output: 2001-03-01\n",
      "-----------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']\n",
    "\n",
    "for example in EXAMPLES:\n",
    "    source = string_to_int(example, Tx, human_vocab)\n",
    "    source = torch.Tensor([source]).long().to(DEVICE)\n",
    "\n",
    "    encoder_output = encoder(source)\n",
    "    sentence = [machine_vocab[\"#\"]]\n",
    "\n",
    "    for t in range(Ty):\n",
    "        prediction = decoder(torch.Tensor([sentence]).long().to(DEVICE), encoder_output)\n",
    "        prediction = torch.softmax(prediction, dim=-1)\n",
    "        prediction = torch.argmax(prediction, dim=-1)\n",
    "        sentence.append(prediction[0][-1])\n",
    "\n",
    "    prediction = prediction.tolist()\n",
    "    #sequential_output = [inv_machine_vocab[s] for s in sentence[1:]]\n",
    "    parallel_output = [inv_machine_vocab[s] for s in prediction[0]]\n",
    "    \n",
    "    print(\"source:\", example)\n",
    "    print(\"parallel output:\", ''.join(parallel_output))\n",
    "    print(\"-----------------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}