notebooks/embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/JohnGiorgi/DeCLUTR/blob/master/notebooks/embedding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q1r5eABrxC3z",
    "tags": []
   },
   "source": [
    "# Embedding text with an existing model\n",
    "\n",
    "This notebook will walk you through embedding some text with a pretrained model using [DeCLUTR](https://github.com/JohnGiorgi/DeCLUTR). You can embed text in one of four ways:\n",
    "\n",
    "1. __SentenceTransformers__: load our pretrained models with the [SentenceTransformers](https://www.sbert.net/) library (_recommended_).\n",
    "2. __🤗 Transformers__: load our pretrained models with the [🤗 Transformers library](https://github.com/huggingface/transformers).\n",
    "3. __From this repo__: import and initialize an object from this repo which can be used to embed sentences/paragraphs.\n",
    "4. __Bulk embed__: embed all text in a given text file with a simple command-line interface.\n",
    "\n",
    "The following pre-trained models are available:\n",
    "\n",
    "- [johngiorgi/declutr-small](https://huggingface.co/johngiorgi/declutr-small)\n",
    "- [johngiorgi/declutr-base](https://huggingface.co/johngiorgi/declutr-base)\n",
    "- [johngiorgi/declutr-sci-base](https://huggingface.co/johngiorgi/declutr-sci-base)\n",
    "\n",
    "Each approach has advantages and disadvantages:\n",
    "\n",
    "1. __SentenceTransformers__: This is the easiest way to use DeCLUTR, but it requires that you [install the SentenceTransformers library](https://www.sbert.net/docs/installation.html) first.\n",
    "2. __🤗 Transformers__: This only requires you to install the [🤗 Transformers library](https://github.com/huggingface/transformers), but requires more boilerplate code.\n",
    "3. __From this repo__: This is the easiest way to use your own models trained with [our repo](https://github.com/JohnGiorgi/DeCLUTR), but requires you to [install DeCLUTR](https://github.com/JohnGiorgi/DeCLUTR#installation).\n",
    "4. __Bulk embed__: This might be suitable if you want to embed large quantities of text \"offline\" (e.g. not on-the-fly within an existing pipeline), but requires you to [install DeCLUTR](https://github.com/JohnGiorgi/DeCLUTR#installation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ObkQs5cixC30"
   },
   "source": [
    "## 🔧 Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iV_jB7IZxC4d"
   },
   "source": [
    "First, let's check to see if we have a GPU available, which we can use to dramatically speed up the embedding of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f5jfZhdRxC4e"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1️⃣ SentenceTransformers\n",
    "\n",
    "Our pretrained models are hosted with Hugging Face Transformers, so they can easily be loaded in SentenceTransformers. Just make sure to [install the SentenceTransformers library](https://www.sbert.net/docs/installation.html) first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -U sentence-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example using [DeCLUTR-small](https://huggingface.co/johngiorgi/declutr-small):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "# Load the model\n",
    "model = SentenceTransformer(\"johngiorgi/declutr-small\", device=device)\n",
    "\n",
    "# Prepare some text to embed\n",
    "texts = [\n",
    "    \"A smiling costumed woman is holding an umbrella.\",\n",
    "    \"A happy woman in a fairy costume holds an umbrella.\",\n",
    "    \"A soccer game with multiple males playing.\",\n",
    "    \"Some men are playing a sport.\",\n",
    "]\n",
    "\n",
    "# Embed the text\n",
    "embeddings = model.encode(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wjg7fRX4xC4s"
   },
   "source": [
    "These embeddings can then be used, for example, to compute the semantic similarity between some number of sentences or paragraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "DvZ3ciiqVXBL",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8356759548187256\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "# Compute a semantic similarity via the cosine distance\n",
    "semantic_sim = 1 - cosine(embeddings[0], embeddings[1])\n",
    "print(semantic_sim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E-mdfiUzVte-"
   },
   "source": [
    "Mainly for fun, the following cells visualize the semantic similarity with a heatmap!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "xsE487dgxC4t"
   },
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "def plot_heatmap(text: List[str], embeddings: np.ndarray) -> None:\n",
    "    embeddings = torch.as_tensor(embeddings)\n",
    "    cosine = torch.nn.CosineSimilarity(-1)\n",
    "    similarity_matrix = []\n",
    "    for _, embedding in enumerate(embeddings):\n",
    "        similarity_vector = cosine(embedding, embeddings)\n",
    "        similarity_vector = similarity_vector.numpy()\n",
    "        similarity_matrix.append(similarity_vector)\n",
    "    df = pd.DataFrame(similarity_matrix)\n",
    "    df.columns = df.index = text\n",
    "    sns.heatmap(df, cmap=\"YlOrRd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "PSc4u6XbUgPD"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAIACAYAAADKXbu0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABZl0lEQVR4nO3dd5xlRZ3//9d7hiQZjAhIElBA4gySVIK4riIqQURFQVZMCOoP96u7StRV0TWhCIhkVhRERUQQJY0IkpkhCgyoIIKg5Dy8f3+cauZO093TDTO3es55Px+P++h76p7wuXXvTH+6qk6VbBMRERER/TWhdgARERERXZQkLCIiIqKCJGERERERFSQJi4iIiKggSVhEREREBUnCIiIiIipIEhYRERExAklHSbpb0jXDvC5J35F0s6SpktYfzXmThEVERESM7BjgzSO8/u/AquWxB/D90Zw0SVhERETECGxfAPxzhF3eDhznxsXAkpKWmd15k4RFREREPD/LAn/t2b69lI1ovrkWTkTEIAdo9ayTVnxy8ozaIYwbS1xwaO0Qxo9H764dwfix1Pv0fA4fy/83+/OnD9N0Iw44wvYRz+f6o5EkLCIiIlpnLAlOSbieT9J1B7B8z/ZypWxE6Y6MiIiI1pkwhscccBrw/nKX5EbA/bbvnN1BaQmLiIiI1pmTrUySfgRsDrxI0u3AfsD8ALYPA84A3gLcDDwC7Daa8yYJi4iIiNaZk0mY7Z1n87qBj4/1vEnCIiIionWe16j+PkkSFhEREa0zLwx6TxIWERERrTMvJDjzQowRERERY5KWsIiIiIgKkoRFREREVJAkLCIiIqKCJGERERERFcwLCc68EGNERETEmKQlLCIiIqKCJGERERERFWTG/IiIiIgK0hIWERERUUGSsIiIiIgK5oUEZ16IMSIiImJM0hIWERERUUGSsIiIiIgK5oUkbK7GKOkdkizpVXPh3C+XdEp5vrmk08vzbSV9dk5f77mS9F8VrrmrpO/2+7rPl6Q/jGHf2yS9aAz7D1snkh4qP5/5Ts3rxlo/Qxy/v6R9yvNjJO0w56KLiJj7JozhUcvcvvbOwO/LzznK9t9sP+sXg+3TbH9lTl/veeh7Ejavsr1J5esP+Z1qK0lpCY+I1up0EiZpUWAzYHfg3cPss4ikX0m6WtI1knYq5bdJ+rKkqyRdJml9SWdJukXSR8o+K0q6ZohzPtPiUf6C/46kP0iaPvDXvKQJkg6VdIOksyWdMdRf+pJeKem3Jb4rJK2ixtdKvNN6Yl5G0gUl5mskvU7SV4AXlLITB8csaR9J+5fn50n6Znm/10uaLOlUSTdJ+mLPMe+TdEk55+GSJpby3ST9SdIlwKbD1Pc0SUuW93CvpPeX8uMkbS1pIUlHl/2ulLRFT53+vNTVbZL2lPTpss/FkpYu+31I0qWlvn4qaeGRPoch4htokXpWXQ61P/CJ8rlMU2ltlbR0iXVqiW3tIa6zkqSLynG9dfvM51Pe86mSziyfwcE9++0+UNeSfqAhWtgkbViucWV536vP7ryDjn+mJUvSJEnnlef7SzpW0hRJf5a0naSDy3s5U9L8Paf5z1J+iaRX9nwWh0n6I3Cwmu/0mZIuL+ccsdVa0r7lM75G0hGS5oX5ECOigyaO4VHL3EwA3w6caftPwL2SNhhinzcDf7O9ju21gDN7XvuL7XWBKcAxwA7ARsABY4xjGZpkcBtgoIVsO2BFYA1gF2DjYY49Efie7XWATYA7y7HrAusAbwS+JmkZ4D3AWSXmdYCrbH8WeNT2urbfO4pYn7A9CTgM+AXwcWAtYFdJL5T0amAnYNNynRnAe8v1D6BJvjYr72soF5Z91gSmAwPJzcbAH8r1bPs1NK2Xx0paqOyzVnnvk4EvAY/YXg+4CHh/2edU25NLfV1Pk4APGOpzGM6z6nKY/e6xvT7wfWCfUnYAcKXttWlaIY8b4rhvA98v7/POEeJYl6a+XwPsJGl5SS8HvkDzXdwUGC5puQF4XamjfYH/Gem8I8QwlFWALYFtgROAc8t7eRR4a89+95fy7wLf6ilfDtjE9qeBI4BP2N6Apg4Pnc21v1s+47WAF9B8nhER48680BI2N7sjdqb5ZQdwUtm+fNA+04D/lfRV4HTbU3peO61nn0VtPwg8KOlxSUuOIY6f234auE7SS0vZZsDJpfzvks4dfJCkxYBlbf8MwPZjpXwz4Ee2ZwB3STqfJjG5FDiqtET83PZVY4hxQO97vtb2neWa04HlS9wbAJeWBogXAHcDrwXOs/2Psv+PgdWGOP8U4PXAn2kSlz0kLQv8y/bD5b0dUt7vDZL+3HOec3s+g/uBX/bEOtDatFZpWVoSWBQ4q+faQ30OwxltXZ5afl5OkyBCU0fbl/dwTkleFx903KYD+wDHA18d5vy/s30/gKTrgBWAFwHn2/5nKT+Zoet6CZokdlXAQG8L1VDn/eswMQzl17aflDSN5o+4gT9eptH8cTHgRz0/v9lTfrLtGWpaqzcBTu5p0FpwNtfeQtJ/AgsDSwPXMvO7EBExbnR2YH7pntoSOFLSbcBngHcN7roorWTr0/zy+KKkfXtefrz8fLrn+cD2WJLH3mPnWteJ7QtoEpw7gGNUuvoGeYpZ63yhQa/P7j0LOLa0rK1re3Xb+48hzAtoWr9eB5wH/IOmhXHKCMcMjm1wfL2fxzHAnqX15QBmfX+j/hxGWZe955zB2P+g8Cj26Y15rNc4iCZxXQt4G8PXxXDn7f2uDPk9KUntk7YH3svgfxse5vnD5ecE4L6e79O6tl893BsqraKHAjuUz/gHQ8Q21HF7qOlmv+wy7pvd7hERc8S80BI2t669A3C87RVsr2h7eeBWZnZ/Ac3daDTdWicAX6NJyPrhQmB7NWPDXgpsPniH0upzu6R3lFgXVDPGaQpNF9JESS+mSRYukbQCcJftHwBHMvO9PNkzTucu4CWldWZBxt6V8ztgB0kvKTEtXa77R+AN5bzzAzsOdbDtv9K05KxqezrNTRP70CRnlPf23nLu1YBXADeOIb7FgDtLDKPpfh3SCHU5Gr3vYXOaLssHBu1zITPHKY41zktp6nopNQPbtx9mvyVokkiAXcd4DYDbaFo9GeEas7NTz8+LBr9Y6uVWSTsCqLHOCOcbSLjuKa1oo7qJwfYRtifZnjSJJUcdfETE8yGN/lHL3ErCdgZ+Nqjspzz7LsnX0CQwVwH7AV+kP34K3A5cRzOm5grg/iH22wXYS9JUmjFTL6N5X1OBq4FzgP+0/XeaRO5qSVfS/NIb6Io9Apgq6UTbTwIHApcAZ9OMGxo129cBnwd+U2I6G1imdFvuT/OL9kKa8VjD+SPwp/J8CrAsTTIGTSvHhNLN9WNgV9uPP/sUw/pCOf+FjPG9DbI5Q9flaOwPbFDq5yvAB4bYZ2/g4+V9LjuWwGzfQTO+6xKa93kbQ393Dga+XN7Dc+n2PwD4tqTLaFrLnoulSj3sDXxqmH3eC+wu6WqarsW3D3cy2/fRtH5dQ9PVfOnAa5I+onLTTETEeDBxgkf9qEUzezK6RdKith+S9EKaX6iblmQqYkQ93535aJLyowbGDsbIDtDq3fwPZwifnPxcc+v2WeKC2d0P0iGP3l07gvFjqfc9rzaqKfOtNur/b1731J+qtId1eZ6g08sA/wWAg5KAxRjsL+mNNN1zvwF+XjeciIgYTBr/f/N1NgmzvXntGGLeZHuf2e8VERE1zQuzGHY2CYuIiIj2ShIWERERUcGEdEdGRERE9N+EtIRFRERE9F+6IyMiIiIqmBfujpwXllaKiIiIGJMJGv1jNCS9WdKNkm6W9NkhXl9B0u8kTZV0nqTlZhvj2N9WRERExPg2J5ctkjQR+B7w78AawM6S1hi029eB42yvTbM6zpdnd94kYREREdE6wqN+jMKGwM22p9t+AjiJZy/ztgbNcoYA5w7x+rMkCYuIiIjWmTBh9A9Je0i6rOexx6DTLQv8tWf7dp699vDVwHbl+TuBxcrSiMPKwPyIiIhonbHME2b7COCI53nJfYDvStoVuAC4AxhxkdgkYREREdE6c3iKijuA5Xu2lytlz7D9N0pLmKRFge1t3zfSSdMdGREREa2jMTxG4VJgVUkrSVoAeDdw2izXk14kaSCv+hxw1OxOmiQsIiIiWkfyqB+zY/spYE/gLOB64Ce2r5V0oKRty26bAzdK+hPwUuBLsztvuiMjIiKidSbO4WYm22cAZwwq27fn+SnAKWM5Z5KwiIiIaJ15Ycb8JGERERHROvPA0pFJwiIiIqJ9soB3RERERAXpjoyIiIioYLQLc9eUJCwiIiJaZ8I8MAlXkrCIiIhonVEuzF1VkrCI6JtPTh5xGbVO+dalE2uHMG7st9CIaxx3y4JL1o6gNTIwPyIiIqICzQODwpKERUREROsoY8IiIiIi+k8T0xIWERER0XdpCYuIiIioQPPAyPwkYREREdE6aQmLiIiIqCEtYRERERH9l5awiIiIiAom5O7IiIiIiP5LS1hEREREDZkxPyIiIqL/5oFx+UnCIiIion2ydmREREREBRkTFhEREVFB1o6MiIiIqCBjwiIiIiIqyJiwiIiIiAoyJiwiIiKihnmgPzJJWERERLTOhPmShEVERET03TzQEJYkLCIiItpnXhiYP6pha5LeIcmSXjXM6ytKumbOhtY+kg6U9Ma5cN69JF0v6cQR9pkk6Ttz+tpjUb5Ha/T5mvtL2meMxzw0TPkxknaYM5HNXZJuk/Si53H8M/U2L73viIhnaAyPSkbbErYz8Pvyc7+5F0672d53Lp36Y8Abbd8+wrUvAy4bXC5pPttPzaW4BnsHcDpwXZ+uFyPo82cfEdFX88LdkbMNUdKiwGbA7sC7R9h1oqQfSLpW0m8kvaAc/yFJl0q6WtJPJS1cyo+RdJikyyT9SdI2pXxXSb+QdJ6kmyTtV8oPlPTJnri+JGnvQbF+RtJe5fk3JZ1Tnm850EokaWdJ0yRdI+mrPcc+JOlrJf7fStqwxDBd0rZlnxUlTZF0RXlsUso3L/ueIukGSSdKz+6N7m1RKC0VB5TzTBuqlXG46w3a5zBgZeDXkj5V4r5I0pWS/iBp9Z4YTy/P95d0vKQLgeMlXSBp3Z5z/l7SOoOuM1HS10u9TZX0iVK+VbnWNElHSVqwlH9F0nVl36+X2LcFvibpKkmrlDqbVPZ/kaTber4DP5d0dqmnPSV9ulznYklLl/1WkXSmpMtLPQ3ZUgus0fNZ7tXznj5d3s81vd+tntcl6buSbpT0W+AlPa/N8v6GOHa4z2FXSaeWuG+SdPBQAaunJUtNK+Z5PZ/dseX9/lnSdpIOLvV/pqT5e07zn6X8EkmvLMcP/Lv7I3DwGOpwIK591fx7vkbSEUN9zyMixgNN0KgftYwmT3w7cKbtPwH3StpgmP1WBb5ne03gPmD7Un6q7cm21wGup0nmBqwIbAi8FThM0kKlfMNy/NrAjuUX9VHA+wEkTaBJCE8YFMMU4HXl+SRg0fJL6XXABZJeDnwV2BJYF5gs6R1l/0WAc0r8DwJfBLYG3gkcWPa5G9ja9vrATkBv9956wCeBNWiSok2Hqade95RzfR8YqstspOsBYPsjwN+ALWx/E7gBeJ3t9YB9gf8Z5tpr0LSe7Qz8ENgVQNJqwEK2rx60/x40n9e6ttcGTiyf1zHATrZfQ9Oy+lFJL6SptzXLvl+0/QfgNOAztte1fcts6mYtYDtgMvAl4JHyni6ifA+AI4BP2N6Apv4OHeZcrwL+jeZ7tZ+k+cv3eDfgtcBGwIckrTfouHcCq5e6ej8wkHQ/6/0Ncc2RPod1aT7P1wA7SVp+NnUx2Co03+Ftaf4NnFvq/1Gaf0sD7i/l3wW+1VO+HLCJ7U8z+joc8N3y73kt4AXANmOMPSKiLyZM1KgfoyHpzeWP8pslfXaI118h6dzyx/dUSW+ZbYyjuO7OwEnl+Ulleyi32r6qPL+c5hc2wFrlL+xpwHuBNXuO+Yntp23fBEyn+WUJcLbte20/CpwKbGb7NpokcD3gTcCVtu8dFMPlwAaSFgcep/mFPYkmCZtC8wv9PNv/KN0wJwKvL8c+AZxZnk8Dzrf9ZHk+8F7mB35Q3svJNL+cB1xi+3bbTwNX9RwzklOHqK9eI11vOEsAJ6sZo/dNZq3vXqeV+qWce5uSsH6QJrEa7I3A4QPdV7b/SZOg3FoSdIBjaerzfuAx4IeStgMeGUXcg51r+0Hb/yjn+2UpnwasqKaFdpPyXq8CDgeWGeZcv7L9uO17aBLbl9K07v7M9sO2H6L5LF436LjXAz+yPcP234BzSvlo3t9In8PvbN9v+zGartkVRlMhPX7d892cyKzf2xV79vtRz8+Ne8pPtj1jjHU4YAtJfyzfyS0Z/vv1DEl7qGnxvuyYux+Y3e4REXPGhDE8ZkPSROB7wL/T/C7eWc8e4/x5mrxmPZqGotn9UTvymLDS7bMl8BpJpvkP35I+Y9uDdn+85/kMmr+SofmF/g7bV0vaFdi8Z7/B5/Bsyo+kabF5GU3L2Kw72U9KurXs8wdgKrAF8EqaVrhVh36nADzZ856eHng/tp+WNFBPnwLuAtah+dge6zl+8PsfzXi7gWOG23+k6w3nIJoE5p2SVgTOG2a/hwee2H5E0tk0rZ7vAoZr7RwV209J2hDYCtgB2JPmezTYU8z8+i806LXe+ny6Z/tpmrqaANxne91RhPRcPpthjfL9jfQ5jCae2dZN+W4O/t72nsvDPB/47MdSh5SWz0OBSbb/Kmn/IWJ7FttH0LS4cf+Grxz8bzsiYu6Ys92MGwI3254OIOkkmt+ZvWOcDSxeni9B00s1coizeX0H4HjbK9he0fbywK08u8VgJIsBd5ZWlvcOem1HSRMkrULThXdjKd9a0tJqxpW9A7iwlP8MeDNNi9ZZw1xvCk23ygXl+UdoWs0MXAK8Qc34o4k0rXrnj+G9LAHcWVq7dqFJSuem53K9JYA7yvNdx3CtI2m6Oy+1/a8hXj8b+PBAQloS9BtpWqVeWfbZBTi/tLAsYfsMmkRyYHzZgzTfhwG3MTPhG9Pdd7YfAG6VtGOJRxo0jm02pgDvkLSwpEVouhenDNrnApruwomSlqFJ6Bnh/fV6rp/DgNuYWTfbj7DfSHbq+XnR4BefQx0OJFz3lDrIHZMRMX6NoSWst8W+PPYYdLZlgb/2bN9eynrtD7xP0u3AGcAnRhPiSHamSXx6/ZThuySH8gXgjzSJ1A2DXvsLTWL0a+AjpXuGUvZTmpasn5Y7+7D9BHAuTXPfjGGuN4WmS+Ui23fRtB5NKcffCXy2nONq4HLbvxjDezkU+ICkq2m6Th+ezf7P13O53sHAlyVdyRhafGxfDjwAHD3MLkfSfF5TSzzvKZ/XbjTdWdNoWmIOo0m0Tpc0leau2k+Xc5wEfKb0l68CfJ1mDNmVwHOZTuG9wO4lnmtp/ioZFdtX0LTSXkLz/TzS9pWDdvsZcBPNXzrHMTORGe799XpOn0OPA4BvS7qMprXsuViqxLg3TbI4lFHXoe37gB8A19D8EXTpwGuSPiLpI88xzoiIOW+CRv2wfYTtST2PI57DFXcGjrG9HPAWmhvfRsyz9Oxexf6QdAxwuu1TBpXvStPdsecQx0wArgB2LOPIYg4pNy2cB7yqtLxFzHHpjpzpW5fO7Yb0ecd+/r/aIYwfw7YvdJA2fF79iTPe/ZpR/38z8aRpI15L0sbA/rb/rWx/DsD2l3v2uRZ4s+2/lu3pwEa27x7uvPPALBqNMgDuZppBzUnA5iBJ76dpDfrvJGAREdEK800Y/WP2LgVWlbSSpAVoBt6fNmifv9CMFUbSq2mGcPxjxBDH/KbmENu7DlN+DEPcnWf7OppxYzGH2T6OprstIiKiHeZgM1O5IWtPmqEYE4GjbF8r6UDgMtunAf8fzYwGn6IZpL/rEDcxziJrR0ZERET7zOFJWMvNWGcMKtu35/l1jG6O0GckCYuIiIj2mQcGXCUJi4iIiPapuBzRaCUJi4iIiPYZ5XJENSUJi4iIiPZJS1hEREREBRkTFhEREVFBWsIiIiIi+m/kBYPGhyRhERER0T5pCYuIiIioYL4kYRERERH9l5awiIiIiAoyJiwiIiKigrSERURERFSQlrCIiIiICtISFhEREVFB7o6MiIiIqCDdkREREREVpDsyIiIiooIkYREREREVpDsyImKmJS44tHYI48Z+C72wdgjjxgF6T+0Qxo19T1mldgjjhrY/4/mdIAPzIyIiIipQkrCIiIiI/hv/OViSsIiIiGihtIRFREREVDD+c7AkYREREdFCaQmLiIiIqGBikrCIiIiI/hv/OViSsIiIiGihdEdGREREVDD+c7AkYREREdFCaQmLiIiIqCAD8yMiIiIqGP852LywxnhERETEGEmjf4zqdHqzpBsl3Szps0O8/k1JV5XHnyTdN7tzpiUsIiIiWmdODgmTNBH4HrA1cDtwqaTTbF83sI/tT/Xs/wlgvdmdNy1hERER0T5ztiVsQ+Bm29NtPwGcBLx9hP13Bn40u5MmCYuIiIj20Rges7cs8Nee7dtL2bMvK60ArAScM7uTpjsyIiIi2mcMd0dK2gPYo6foCNtHPMcrvxs4xfaM2e2YJCwiIiLaZwyDwkrCNVLSdQewfM/2cqVsKO8GPj6a66Y7MiIiItpnznZHXgqsKmklSQvQJFqnPeuS0quApYCLRnPSJGERERHRPnNwYL7tp4A9gbOA64Gf2L5W0oGStu3Z9d3ASbY9mhDTHRkRERHtM4cna7V9BnDGoLJ9B23vP5ZzJgmLiIiI9pkw/qfM71x3pKR3SHLpt43nSdIZkpYsj4/1lG8u6fQ+XP8YSTuU55+UtPDg2GZz/HmSJs3lMHuvd5ukF82lcz/nOpe07VAzQEdEzLMmaPSPWiFWu3I9OwO/Lz/HHUnzVOuk7bfYvg9YEvjYyHvPdZ8EnknCemKL2bB9mu2v1I4jImKOmbMD8+eKTiVhkhYFNgN2pxk8N9Q+i0j6laSrJV0jaadSvpWkKyVNk3SUpAVL+WRJfyj7XyJpMUkTJX29HD+1LF+ApA0knS/pcklnSVqmlJ8n6VuSLgP2HhTPiyWdLelaSUdK+vNAS4qkn5dzXVvmOBk45iFJXyvlv5W0YbnG9IEBhCXGr0m6tMT44SHq4jOS9irPvynpnPJ8S0knlucDLTtfAVYpa2Z9rZxiUUmnSLpB0onSs0c/lri+KekySdeX+jxV0k2Svlj2WVHSNT3H7CNp/0Hn2Qt4OXCupHN7YyvHD8RwfYlpYQaR9CZJF0m6QtLJ5fsy5nhH+mwGnet95TtzlaTDy2cyUU3r3jXlu/apIY47RtJhJYY/SdpmiH02LO/lyvL9XL2UXyBp3Z79fi9pHUm7Svpuz/m/U46brpktjRMkHVrq8mw1LY07DPXeIiKqm8NrR84NnUrCaJYYONP2n4B7JW0wxD5vBv5mex3bawFnSloIOAbYyfZraMbSfVTNbao/Bva2vQ7wRuBRmgnfVgTWtb02cKKk+YFDgB1sbwAcBXyp57oL2J5k+38HxbMfcI7tNYFTgFf0vPbBcq5JwF6SXljKF+k55kHgizTrXb0TOLDssztwv+3JwGTgQ5JWGnTtKcDryvNJNEnV/KXsgkH7fha4xfa6tj9TytajaZ1aA1gZ2JShPWF7EnAY8Aua+VXWAnbteU8jsv0d4G/AFra3GGKX1YFDbb8aeIBBrXYlkfw88Ebb6wOXAZ9+HvEO99kMXO/VwE7AprbXBWYA7wXWBZa1vVb5rh09TAwr0iyj8VbgsPId7XUD8Drb6wH7Av9Tyn8I7FpiWA1YyPbVQ5x/GZo/WLahSbABtivXXQPYBdh4mNgiIupLS9i4szPNek+Un0N1SU4Dtpb0VUmvs30/zS/wW0vyBnAs8PpSfqftSwFsP1BuY30jcHh5ju1/ln3XAs6WdBXNL/zleq7742Fi3mwgZttnAv/qeW0vSVcDF9NMIrdqKX8COLPn/Zxv+8nyfMVS/ibg/SWWPwIv7Dl+wOXABpIWBx6nmfdkEk0SNmWYeHtdYvt2208DV/Vce7CBuVamAdfavtP248B0Zp0c7/n4q+0Ly/MTaOq110Y0ycWFpU4+AKzwPOId7rMZsBWwAc0isFeV7ZXLOVaWdIikN9MkjEP5ie2nbd9Ujhk8xnEJ4OTSgvhNYM1SfjKwTUmmP0jzx8VQfl7Ofx3w0lK2GXByKf87cO4wx0ZE1DcPjAmbp8YfPR+Slga2BF4jycBEwJI+0zufh+0/SVofeAvwRUm/o2nteN4h0PzCHq714OExnUzanCbZ29j2I5LOAwZaQ57seU9P0yRQ2H5aM8ecCfiE7bOGu4btJyXdStNy8gdgKrAF8EqaeVJm5/Ge5zMY/vs2sN/TzHrM0+WYp5j1D4bBrT6jMXjOlsHbAs62PZqxgiPGO5vPpvd6x9r+3OCTS1oH+DfgI8C7aJKlwWb3fg4CzrX9TkkrAucBlHjOpmkVfhdNIjjSexyI9TlTz3Igh393b/bY/S3P53QREaMzYfy3M43/COecHYDjba9ge0XbywO3MrO7DQBJLwcesX0C8DVgfeBGYEVJryy77QKcX8qXkTS5HLtYSXLOBj48kPCUBPBG4MWSNi5l80tak9m7kOaXJZLeRDMTLzQtHf8qv1RfRdOSMxZn0XSpzl/OvZqkRYbYbwqwD0334xSaxODKISaiexBYbIwxjNZdwEskvVDNWLxnjYEaRQyvGKh74D00N2f0uhjYdOAzVjM2cLXnGO9oPpvfATtIekm53tKSVijdohNs/5SmtXT9Ya6xYxmjtQpNC9qNQ8QwsKTGroNeOxL4DnCp7X8xehcC25frvhTYfDQH2T6idLVPSgIWEX2TMWHjys7AzwaV/ZRnd0m+BrikdBHtB3zR9mPAbjTdO9NoWjwOs/0EzbieQ0rX09k0LR5HAn8Bppby95R9dwC+WsquAjYZRdwHAG8q3Uo7An+nSTbOpGl1uZ5mzM7Fo62I4kjgOuCKcu7DGbqlagrN+KCLbN8FPMYQXZG276XpyrtGMwfmzxGlK/VA4BKaOr5hmF2PoBnDN1Q32Y3Ax0t9LQV8f9A1/kGTrPxI0lSartfnOo3JbD+b0s33eeA35Xpn09TzssB55ft3AvCslrLiLzT18WvgI+U72utg4MuSrmTQ52r7cppuzuHGmw3np8DtNN+bE4ArgPsB9OxZoyMi6tKE0T9qhTjKmfWjktLyM8P2U6Ul5/tlIHeMUumOO73caDHPk3QMzfs55Tke/3Ka7slXlfF6Yzl2UdsPlRsNLqG5seDvoz7BY7/JfzgDFhrVPSedcIDeUzuEcWPfU1apHcK4oe3PeF5NVD5r+1H/f6N/+2mV5rDOjAmbh70C+ImkCTQD7j9UOZ6Yh0l6P81duZ8eawJWnK5mAtwFgIPGlIBFRPRTxW7G0UoSNs6Vu9/Wqx3HvMz2bTR3praC7V2fx7HHAcc9j+M3f67HRkT0VcVuxtFKEhYRERHtMzFJWERERET/pSUsIiIiooKMCYuIiIioIElYRERERAXpjoyIiIiooOKakKOVJCwiIiLaZ8LE2hHMVpKwiIiIaJ+MCYuIiIioIN2RERERERVkYH5EREREBemOjIiIiKhgQlrCIiIiIvovSVhEREREBemOjIiIiKggSVhEREREBbk7MiIiIqKCzBMWERERUUGWLYqIiIioIGPCIiIiIipIEhYRERFRQeYJi4jo8ejdtSMYPxZcsnYE48a+p6xSO4Rx48Adbqkdwrixn5/vGeZsS5ikNwPfBiYCR9r+yhD7vAvYHzBwte33jHTOJGERERHRPnOwO1LSROB7wNbA7cClkk6zfV3PPqsCnwM2tf0vSS+Z3XmThEVERET7aI7eHbkhcLPt6QCSTgLeDlzXs8+HgO/Z/heA7dk2/Y//DtOIiIiIsZJG/5i9ZYG/9mzfXsp6rQasJulCSReX7ssRpSUsIiIi2mcMM+ZL2gPYo6foCNtHjPGK8wGrApsDywEXSHqN7ftGOiAiIiKiZUY/JqwkXCMlXXcAy/dsL1fKet0O/NH2k8Ctkv5Ek5RdOtxJ0x0ZERER7TNnuyMvBVaVtJKkBYB3A6cN2ufnNK1gSHoRTffk9JFOmpawiIiIaJ85ODDf9lOS9gTOopmi4ijb10o6ELjM9mnltTdJug6YAXzG9r0jnTdJWERERLTPHJ4x3/YZwBmDyvbteW7g0+UxKknCIiIiooWybFFERERE/43h7shakoRFRERE+2QB74iIiIgakoRFRERE9J0mzNFli+aKJGERERHRQmkJi4iIiOi/DMyPiIiIqCEtYRERERH9l7sjIyIiIipId2REREREBUnCIiIiIipIEhYRERFRw/gfEzb+08SYoyT9t6RrJU2VdJWk19aOaV4jaUVJ1zzHYydJ+s6cjqnfJO0q6eW144iIGJY0+kclaQnrEEkbA9sA69t+XNKLgAUqh9V3kuaz/VSNa9u+DLisxrXnFEkTgV2Ba4C/1Y0mImI447+dafxHGHPSMsA9th8HsH2P7b8BSNpK0pWSpkk6StKCpfw2SV8urWaXSVpf0lmSbpH0kYETS/qMpEtLC9sBQ11c0kOSvlZa4n4raUNJ50maLmnbss/Ess/AuT5cyjcv+54i6QZJJ0rP/vNF0ofKsVdL+qmkhUv5MZIOk/RH4GBJq0g6U9LlkqZIetUQ59pf0vGSLpJ0k6QPDbHPiuX4K8pjk1J+nKR39Ox3oqS3l/dxes/5j+qpg7169v+CpBsl/V7SjyTtM8S13ybpj+Vz+62klw6xz5qSLimf31RJq5aYB+rw+lKnA/U00vfgq5KuAHYGJgEnlvO+YKjPOyKiqgkTRv+oFWK1K0cNvwGWl/QnSYdKegOApIWAY4CdbL+GpoX0oz3H/cX2usCUst8OwEbAAeX4NwGrAhsC6wIbSHr9ENdfBDjH9prAg8AXga2BdwIHln12B+63PRmYDHxI0krltfWATwJrACsDmw5xjVNtT7a9DnB9Od+A5YBNbH8aOAL4hO0NgH2AQ4eps7WBLYGNgX2H6IK7G9ja9vrATsBAV+MPaVqLkLQEsAnwqyHO/yrg32jqbj9J80uaDGwPrAP8O03CM5TfAxvZXg84CfjPIfb5CPDt8vlNAm4v5asDh9p+NfAA8LFRfA/utb2+7RNoWvPea3td248OE19EREUTxvCoI92RHWL7IUkbAK8DtgB+LOmzwJXArbb/VHY9Fvg48K2yfVr5OQ1Y1PaDwIOSHpe0JPCm8riy7LcoTVJ2waAQngDO7DnX47aflDQNWLGUvwlYW9IOZXuJcq4ngEts3w4g6apyzO8HXWMtSV8ElixxnNXz2sm2Z0halCYpOrmnMW3BZ9cYAL8oScajks6lSZau6nl9fuC7ktYFZgCrAdg+vyS6L6ZJqH5q+6khGu9+VVomH5d0N/BSmuTyF7YfAx6T9MthYluO5jNchqZb+dYh9rkI+G9Jy9EkqDeVGP5q+8KyzwnAXsDZjPw9+PEwcUREjD/zwGStaQnrGNszbJ9nez9gT5oEYXYeLz+f7nk+sD0fzS0oXy6tIuvafqXtHw5xnidte/C5bA+ch3KuT/ScayXbvxkUBzQJz1B/RBwD7Flacg4AFup57eHycwJwX8811i0tQkPxbLY/BdxF02o1iVnH2B0HvA/YDThqmPOP5j0N5xDgu+W9fphZ32sTrP1/wLbAo8AZkrYc5n0M3h7Kw7Pf5dkk7aGmK/uyI44557mcIiLiOdAYHnUkCesQSatLWrWnaF3gz8CNwIqSXlnKdwHOH8OpzwI+WFqYkLSspJc8xzDPAj4qaf5yrtUkLTKG4xcD7izHv3eoHWw/ANwqacdyDUlaZ5jzvV3SQpJeCGwOXDro9SWAO0siuQswsee1Y2i6T7F93Rjew4XA28p1F6W5mWIoSwB3lOcfGGoHSSsD021/B/gFTfcqwCvU3KgB8B6aFsWxfA8epKnr2bJ9hO1JtiftseuWsz8gImJO0ITRPypJEtYtiwLHSrpO0lSasVX7l26v3Wi656bRtFIdNtqTlpaq/wMuKsefwih/QQ/hSOA64Ao100Aczthah74A/JEmkblhhP3eC+wu6WrgWuDtw+w3FTgXuBg4aOBGhh6HAh8o53kVPa1Ftu+iGZd29Bjix/alNF3AU4Ff03Td3j/ErvvTfGaXA/cMc7p3AdeU7tu1aFrnoEm4Pi7pemAp4Ptj/B4cAxw2MDBf0oEqN1dERIwL88AUFZrZOxQRvSTtDzxk++vP8fiFaRKo9W0PlUSNdOyiZQzfwjRj6/awfcVziWOIc68InG57rTlxvjH51wn5D2fAkqvVjmDc8Kn71w5h3Dhwh1tqhzBu7Ocbn192dP9Jo///Zol3V8nE0hIWMRdIeiNNK9ghY03AiiNK69UVNIP650gCFhHRHeN/TFjujowYhu39n8exvwVWeB7Hv+e5HjuKc99G0zUZEdFe88DdkUnCIiIiooXGf2dfkrCIiIhon7SERURERFSgibPfp7IkYREREdE+aQmLiIiIqCFjwiIiIiL6Ly1hERERETWkJSwiIiKi/9ISFhEREVHBPHB35Phvq4uIiIgYqzm8gLekN0u6UdLNkj47xOu7SvqHpKvK4z9md860hEVEREQLzbnuSEkTge8BWwO3A5dKOs32dYN2/bHtPUd73rSERURERPtowugfs7chcLPt6bafAE4C3v58Q0wSFhERES2kUT8k7SHpsp7HHoNOtizw157t20vZYNtLmirpFEnLzy7CdEdGRERE+4yuhQsA20cARzzPK/4S+JHtxyV9GDgW2HKkA9ISFhEREe0zZ7sj7wB6W7aWK2XPsH2v7cfL5pHABrM7aZKwiIiIaJ85m4RdCqwqaSVJCwDvBk6b5XLSMj2b2wLXz+6k6Y6MiIiIFppzd0fafkrSnsBZwETgKNvXSjoQuMz2acBekrYFngL+Cew6u/MmCYuIiIj2mcMz5ts+AzhjUNm+Pc8/B3xuLOdMEhYREREtNP5HXCUJi4iIiPaZB5YtShIWERER7TMPLOAt27VjiIjoG0l7lDmBOi91MVPqYqbURf+M/w7TiIg5a/BM2F2WupgpdTFT6qJPkoRFREREVJAkLCIiIqKCJGER0TUZ6zJT6mKm1MVMqYs+ycD8iIiIiArSEhYRERFRQZKwiIiIiAqShEVERERUkCQsIjpHUuZBKlIXM0napnYM40Xqoj+ShEVEF43/9Uz6J3Ux0+TaAYwjqYs+yN2RERERHSZpQduPz64s5rws4B0RrSfprcCawEIDZbYPrBdRPamLhqTthii+H5hm++5+x1PZRcD6oyiLOSxJWES0mqTDgIWBLYAjgR2AS6oGVUnqYha7AxsD55btzYHLgZUkHWj7+FqB9YuklwHLAi+QtB4zu6YXp/mexFyW7siIaDVJU22v3fNzUeDXtl9XO7Z+S13MJOks4P227yrbLwWOA3YGLrC9Vs34+kHSB4BdgUnApcxMwh4AjrV9aqXQOiMtYRHRdo+Wn49IejlwL7BMxXhqSl3MtPxAAlbcXcr+KenJWkH1k+1jJR0P7Gz7xNrxdFGSsIhou9MlLQl8DbgCME1XXBelLmY6T9LpwMlle/tStghwX7Wo+sz205I+BSQJqyDdkRHRGZIWBBayfX/tWGrrel1IEk3itWkpuhD4qTv4S1HSV4B7gB8DDw+U2/5ntaA6IklYRLTSMHe/PaNL411SFzESSbcOUWzbK/c9mI5Jd2REtNXbRnjNQJcSj9TFICUx/SrwEpoB6aJJPBavGlgFtleqHUNXpSUsIiI6R9LNwNtsX187ltokzQ98FHh9KToPONx2J25QqClJWES0kqRPj/S67W/0K5baUhfPJulC25vOfs/2k3QkMD9wbCnaBZhh+z/qRdUN6Y6MiLZarHYA40jq4tkuk/Rj4OfAM8vzdHR83GTb6/RsnyPp6mrRdEhawiIionMkHT1EsW1/sO/BVCbpCmBH27eU7ZWBU2xn2aK5LElYRLSapNWA7wMvtb2WpLWBbW1/sXJofZe6iKFI2go4GphOc4PCCsButs8d8cB43pKERUSrSTof+AzNQOP1Stk1XViWZrDUBUj6T9sHSzqE5s7QWdjeq0JY1ZV541Yvmzfafnyk/WPOyJiwiGi7hW1f0szN+YynagVTWeoCBu6GvKxqFOOIpIWAjwGb0SSmUyQdZvuxupG1X5KwiGi7eyStQmn1kLQDcGfdkKrpfF3Y/mV5eoXtaVWDGT+OAx4EDinb7wGOB3asFlFHpDsyIlqtDDI+AtgE+BdwK/Be23+uGlgFqYuZJE0BFgSOAU7s6vJNAJKus73G7MpizktLWES0lqSJwMdsv7EszDzB9oO146ohdTEr26+TtCrwQeBySZcAx9j+TeXQarhC0ka2LwaQ9FrSXdsXaQmLiFaTdLHtjWrHMR6kLp6tJKfvAL4DPEBzd+B/dWm+MEnX0wzK/0spegVwI814Qdteu1ZsbZckLCJaTdL3gWWBk4GHB8q79Et2QOpipjI9x27AW4GzgR/avkLSy4GLbK9QNcA+kjTie+1id3W/pDsyItpuIeBeYMuesk4uWk3qotchwJE0rV6PDhTa/pukz9cLq4r5gNttPy5pc2Bt4Djb99UMqgvSEhYREdFhkq4CJgErAmcAvwDWtP2WimF1woTaAUREzE2SVpb0S0n/kHS3pF9IWql2XDWkLmaStKqkUyRdJ2n6wKN2XJU8bfspYDvgENufAZapHFMnJAmLiLb7P+AnNL9UXk4zHuqkqhHVk7qY6WiaJZyeAragmSvrhKoR1fOkpJ2B9wOnl7L5K8bTGUnCIqLtFrZ9vO2nyuMEmrFRXZS6mOkFtn9HMyznz7b3pxmk30W7ARsDX7J9a2kdPb5yTJ2QMWER0UqSli5P/x/NxKQn0QxC3wlYyvbnasXWb6mLZ5P0B5plek4BzgHuAL5ie/URD4yYg5KERUQrSbqVJtHQEC/b9sp9Dqma1MWzSZpMs47kksBBwBLAwQMTlkb0Q5KwiIiIiAoyT1hEtFqZEf2tNLffP/N/nu1v1IqpltQFSPolZQHzodjeto/hRMclCYuItvsl8BgwDXi6ciy1pS7g67UDGG8kvZhmvOAa9NyoYXvLYQ+KOSJJWES03XJZ++4Zna8L2+cPPJe0APAqmpaxG20/US2wuk4EfkzTSvoR4APAP6pG1BGZoiIi2u7Xkt5UO4hxInVRSHorcAvNwt3fBW6W9O91o6rmhbZ/CDxp+3zbH2TWpa1iLklLWES03cXAzyRNAJ6kuUPQthevG1YVqYuZ/hfYwvbNAJJWAX4F/LpqVHU8WX7eWZLTvwFLj7B/zCFJwiKi7b5BMxHlNOd28NTFTA8OJGDFdODBWsFU9kVJSwD/H83C5osDn6obUjdkioqIaDVJFwCb2+7qQPRnpC5mkvR9YAWaZZwM7Aj8BfgtgO1T60UXXZEkLCJaTdIxwMo03UyPD5R3aVqGAamLmSQdPcLLLuOiIuaqdEdGRNvdWh4LlEeXpS4K27vVjiEiLWERERERFaQlLCJaTdK5DDFDehcnokxdxFAkfXqI4vuBy21f1edwOiVJWES03T49zxcCtgeeqhRLbamLGMqk8vhl2d4GmAp8RNLJtg+uFlnLpTsyIjpH0iW2N6wdx3jQ1bqQtDdwNM20FEcC6wGftf2bqoFVUO6afYvth8r2ojRzpr2ZpjVsjZrxtVlawiKi1ST1Tjo5AdgAWKJSOFWlLmbxQdvflvRvwFLALsDxQOeSMOAl9NwtSzN560ttPyrp8WGOiTkgSVhEtN3lNOOgRNP1diuwe9WI6kldzKTy8y3A8bavlaSRDmixE4E/SvpF2X4b8H+SFgGuqxdW+6U7MiIiOqfME7YssBKwDjAROM/2BlUDq0TSZGCTsnmh7ctqxtMVScIiIqJzyvqZ6wLTbd8n6YXAsran1o2sDkkTgZfS00Nm+y/1IuqGdEdGREQXGViD5k7AA4FFaO4Y7RxJnwD2A+4CZlAWdgfWrhlXF6QlLCIiOqesHfk0sKXtV0taCviN7cmVQ+s7STcDr7V9b+1YuiYtYRHRepKWpVmsuber5YJ6EdWTunjGa22vL+lKANv/ktTVpZz+SjM5a/RZkrCIaDVJXwV2ornLa0YpNtC5xCN1MYsnyzgoA0h6MU3LWBdNB86T9Cs6vrB7vyUJi4i2ewewuu3Md5S66PUd4GfASyR9CdgB+HzdkKr5S3l0fmH3fksSFhFtNx2Yn1kno+yq1EVh+0RJlwNb0QxEf4ft6yuHVYXtA2rH0FVJwiKi7R4BrpL0O2btatmrXkjVdL4uBq0acDfwo97XbP+z/1HVIelbtj8p6ZcMvbD7thXC6pQkYRHRdqeVR6QuYNZVAwYMbBtYuUZQlRxffn69ahQdlikqIiIiOkzS24Bf2e7qjQnVJAmLiFaTtCrwZZqJOZ+ZjNN2l1o8gNTFYGVusFWZtS46d6eopBOAjYGfAkfZvqFySJ0xoXYAERFz2dHA92kWrN4COA44oWpE9aQuCkn/QTM1x1nAAeXn/jVjqsX2+4D1gFuAYyRdJGkPSYtVDq31koRFRNu9wPbvaFr+/2x7f+CtlWOqJXUx097AZODPtregSULuqxpRRbYfAE4BTgKWAd4JXFGWNIq5JAPzI6LtHi+LNd8kaU/gDmDRyjHVkrqY6THbj0lC0oK2b5C0eu2gapC0LbAb8Eqa1tENbd8taWGaiX0PqRlfmyUJi4i22xtYGNgLOIimG+4DVSOqJ3Ux0+2SlgR+Dpwt6V/An6tGVM/2wDcHj4ez/Yik3SvF1AkZmB8REZ0m6Q3AEsCZtp+oHU90R8aERUREJ0laStLawIPA7cBalUOqQtJGki6V9JCkJyTNkPRA7bi6IN2RERHROZIOAnalWcppYH4sA1vWiqmi7wLvBk4GJgHvB1arGlFHpDsyIiI6R9KNwGvS/QiSLrM9SdJU22uXsittr1c7trZLS1hEtJqk7wxRfD9wme1f9DuemiQdDHwReBQ4E1gb+JTtLs4Vdg2wJM36kV33iKQFaNYVPRi4kwxX6otUckS03ULAusBN5bE2sBywu6Rv1QurijeV+aC2AW6jmZLgM1UjqufLwJWSzpJ02sCjdlCV7AJMBPYEHgaWp7ljMuaytIRFRNutDWxqewaApO8DU4DNgGk1A6tg4P/8twIn275f0kj7t9mxwFdpvgOdXjPR9sDUHI/SrB4QfZIkLCLabimaCUnvL9uLAEvbniHp8XphVXG6pBtoftl+VNKLgccqx1TLI7aH6qruDEnTaG5GGNLA+LCYezIwPyJarUw2+XngPEDA64H/AX4E7G+7U91xkpYG7i9J6CLAYrb/XjuufpP0DeBx4LTyEwDbV1QLqs8krTDS6z0tZDGXJAmLiNaTtAywYdm81PbfasbTb5K2G+l126f2K5bxQtK5QxTbdhenqEDSy2j+jZjm30jnEvMakoRFROtJWhZYgZ4hGIOXaGkzSUeXpy8BNgHOKdtbAH+wvU2VwGJckPQfwL403wsBbwAOtH1U1cA6IElYRLSapK8COwHX0jMpp+1t60VVh6TfAB+wfWfZXgY4xva/1Y0saipzpm1i+96y/UKa5LyTC5r3UwbmR0TbvQNY3XbXBuEPZfmBBKy4C3hFrWBi3LiXZummAQ+WspjLkoRFRNtNB+anZ/B1h/1O0lk0NyVA00L424rxxPhwM/BHSb+gGRP2dmCqpE8D2P5GzeDaLElYRLTdIzQzgf+OWe+C26teSHXY3rMM0n9dKTrC9s9qxlSLpIWB/w94he0PSVqVpsX09Mqh1XBLeQwYWElisQqxdErGhEVEq0n6wFDlto/tdywxfkj6MXA58H7ba5Wk7A+2160bWXRJkrCIiJaT9CBDT8opmpsUFu9zSNX1LFr9zELVkq62vU7t2KI70h0ZEa0k6Se23zXcrOBdmg3cdrqVnu0JSS+gfDckrULGDUafpSUsIlpJ0jK27xxuVvCuzgYuaR1mjgm7wPbUmvHUImlrmpUU1gB+A2wK7Gr7vJpxRbckCYuI6AhJewMfAgZmyH8nzeD8Q+pFVU+ZD2sjmm7Zi23fUzmkKiStBnwfeGkZH7c2sK3tL1YOrfWShEVEK40wDgqAjo6DmgpsbPvhsr0IcFGXumYlrT/S611aO3KApPOBzwCH94yPu8b2WnUja7+MCYuIVhoYByXpIOBO4HiaFo/3AstUDK0mATN6tmeUsi753xFeM9DFtSMXtn2JNMtX4alawXRJkrCIaLttB93x9n1JV9Osldc1R9NMyjkwN9g7gB/WC6f/bG9RO4Zx6J5yY8LATQo70PzhEnNZkrCIaLuHJb0XOInml8zOwMN1Q6rD9jdK19OmpWg321fWjKkWSQsBHwM2o/leTAEOs/1Y1cDq+DhwBPAqSXcAt9K0GMdcljFhEdFqklYEvk2TeBi4EPik7dsqhlWNpInAS+n5I9z2X+pFVIekn9CskXhCKXoPsKTtHetF1X/l+/BV2/uUMYITbD84u+NizkgSFhHREZI+AexHs3D3wHgwd2lg/gBJ19leY3ZlXSDpYtsb1Y6ji9IdGRGtJOkQRr47snNrRwJ706yPeG/tQMaBKyRtZPtiAEmvBS6rHFMtV0o6DTiZnq5626cOf0jMCUnCIqKtuvoLdSR/Be6vHcQ4sQHwB0kDXbGvAG4cWGGhY62DCwH3MuudoWbmfHIxl6Q7MiI6QdKiALYfqh1Lv0n6dHm6JrA68Ct6luix/Y0acdU03EoKA7q6okL0V1rCIqLVJK1FM0fY0s2m/gG83/a1dSPrq4G1I/9SHguUR+dIWtz2AzSD8p/F9j/7HFJ15U7R3WmS9IUGym1/sFpQHZGWsIhoNUl/AP7b9rlle3Pgf2xvUjOuqEPS6ba3kXQrTZdb7wyltr1ypdCqkXQycAPNHaIH0kxPcb3tvasG1gFJwiKi1SRdPWiy1iHLIrpK0pW215M01fbakuYHpuSOyblvQu0AIiLmsumSviBpxfL4PDC9dlBRl6TfjaasI54sP+8r3fdLAC+pGE9nZExYRLTdB4EDmHmn15RSFh1Uxj8tDLxI0lLM7I5cHFi2WmB1HVHq4vPAacCiwBfqhtQN6Y6MiOgISasB3wdeanstSWvTrK35xcqh9Y2kvYFPAi8H/tbz0gPAD2x/t0Zc0U1JwiKi1UrisQ+wIrMu1bPlcMe0VVk38jPA4bbXK2XX2F6rbmT9J+kTtg+pHUd0W7ojI6LtTgYOA46kWaqnyxa2fYnUe0MgT9UKpgZJ25Wnd/Q8f0ZmiY9+ShIWEW33lO3v1w5inLhH0iqU5Zwk7QDcWTekvnvbCK9llvjoq3RHRkQrSVq6PN0LuBv4GbPOEt/FSTlXBo4ANgH+BdwKvM/2bTXjivokbcKzu+yPqxZQRyQJi4hWGmYyzgGdnJRzgKRFgAm2h5w1vgsk7TtUue0D+x1LbZKOB1YBrmJml707ush9X6U7MiJayfZKtWMYbyQtCbyf0uIxMDaso79sH+55vhCwDXB9pVhqmwSs4bTK9F2SsIiI7jgDuBiYBjxdOZaqbP9v77akrwNnVQqntmuAl9G98YHVJQmLiOiOhWx/unYQ49TCwHK1g6jkRcB1ki5h1nGT29YLqRuShEVEdMfxkj4EnE5uUphGuUsUmAi8mGbx6i7av3YAXZWB+RHRapI2Ba6y/bCk9wHrA9+2/efKofWdpI8DXwLuY2YC0smbFCSt0LP5FHCX7U7Nmdar1Meqtn8raWFgYpdv3OiXJGER0WqSpgLrAGsDx9BM2vou22+oGVcNkqYDG9q+p3Ys40FZL3F5Zp2W4Yp6EdVRWkf3AJa2vYqkVYHDbG9VObTWS3dkRLTdU7Yt6e3Ad23/UNLutYOq5GbgkdpBjAeSDgJ2BW6hp1UQ6NxyVsDHgQ2BPwLYvknSS+qG1A1JwiKi7R6U9DngfcDrJU0A5q8cUy0PA1dJOpdZx4R1cYqKdwGr2H6idiDjwOO2nxiYskTSfMxMTGMuShIWEW23E/AeYHfbf5f0CuBrlWOq5eflEc20DEvSrKbQdedL+i/gBZK2Bj4G/LJyTJ2QMWEREdE5kiYBv6BJxjo9LUNpHd4deBPNChNnAUdm8ta5L0lYRLSSpN/b3kzSg8zatSKaOwIXrxRaNT1LOc2io3dHXgsczqCJa22fXy2o6Jx0R0ZEK9nerPxcrHYs48iknucLATsCSw+zb9s9Yvs7tYMYDyRtAxwErECTF3T2D5V+S0tYRESHSbrc9ga14+g3Sd+g6YY8jVm7I7s4RcXNwHbAtHRB9ldawiIiOkLS+j2bE2haxrr6e2C98nOjnrKuTlHxV+CaJGD9l5awiIiOKFNTDHgKuBX4X9s3VgopxgFJk2m6I89n1lbBb1QLqiO6+hdQREQX7W57em+BpJVqBRPjxpeAh2jGCS5QOZZOSUtYRLSapO2ArwIvoRlw3NlBx5KusL3+oLJOjgmLmSRdY3ut2nF0UVrCIqLtDgbeZvv62oHUIulVwJrAEiUpHbA4TetHdNsZkt5k+ze1A+maJGER0XZ3dTkBK1YHtqGZIf5tPeUPAh+qEdB4IGkTYEVmXcD7uGoB1fNRYB9JjwNP0uHW4n5Ld2REtJqkbwMvo1mup3fQ8am1YqpF0sa2L6odx3gg6XhgFeAqYEYpdkfX0YxK0hIWEW23OPAIzZIsAwx0LgkD3llmin8UOBNYG/iU7RPqhlXFJGCNTMsQNaUlLCKiIyRdZXtdSe+k6Z78NHCB7XUqh9Z3kk4G9rJ9Z+1YorvSEhYRrSZpIZrFidekZxC67Q9WC6qe+cvPtwIn275fUs14+k7SL2laQhcDrpN0CR1fwDvqSRIWEW13PHAD8G/AgcB7ga4O1P+lpBtouiM/KunFwGOVY+q3r9cOYDyStBmwqu2jy/diUdu31o6r7dIdGRGtJulK2+tJmmp7bUnzA1NsbzTbg1tI0tLA/bZnSFoYWNz232vH1W+Svmr7/82urAsk7UczRm5126tJejlNS+mmlUNrvQm1A4iImMueLD/vk7QWsATNxK2dUxLQ9wE/lnQKTTftvXWjqmbrIcr+ve9RjA/vBLYFHgaw/Tea7tqYy9IdGRFtd4SkpYAvAKcBiwL71g2pmu/TjAs7tGzvUsr+o1pEfSbpo8DHgJUlTe15aTHgwjpRVfeEbUsygKRFagfUFemOjIjoCElXD74TcqiyNpO0BLAU8GXgsz0vPWj7n3WiqkvSPsCqNK2DXwY+CPyf7UOqBtYBScIiotUkLQm8n2fPjN65STklXQHsaPuWsr0ycMrg9STbTNLith8oY+OepcOJ2NY0c+kJOMv22ZVD6oQkYRHRapL+AFwMTAOeHii3fWy1oCqRtBVwNDCd5pftCsButs+tGlgfSTrd9jaSbqWZqqJ3jg7bXrlSaNVJWpxZ/1DpZELaT0nCIqLVJF3RpZae2ZG0IM1akgA32n58pP2j/SR9GDiAZrqSp5m5dmRnE9J+SRIWEa0m6VPAQ8DpzDopZ+f+ypf0ceBE2/eV7aWAnW0fOuKBLVTWjryAZrqSG2rHU5Okm4CNbd9TO5auyRQVEdF2TwBfAy4CLi+Py6pGVM+HBhIwANv/Aj5UL5yqjgKWAQ6RNF3STyXtXTuoSm6hWV81+iwtYRHRapKmAxvmr3yQNA1Ye2DRakkTgam216wbWR3l/U8GtgA+Ajxq+1V1o+o/SevRjBX8I7O2Fnfu5pV+yzxhEdF2N5O/8gecSTNR6+Fl+8OlrHMk/Q5YhKaFdAow2fbddaOq5nDgHAbdvBJzX5KwiGi7h4GrJJ1L/sr/f8AewEfL9tnAkfXCqWoqsAGwFnA/zYoKF9l+tG5YVcxv+9O1g+iidEdGRKtJ+sBQ5V2coiKeTdJiwK7APsDLbC9YN6L+k/Q/wG3AL+n4zSv9liQsIlpP0gLAamXzRttPjrR/tJ+kPYHX0bSG3UbTJTnF9jk146qhzJk2WKao6IN0R0ZEq0naHDiW5hetgOUlfcD2BRXDivoWAr4BXG77qdrB1GR7pdoxdFVawiKi1SRdDrzH9o1lezXgR7Y3qBtZ/0l6je1pteOI8UXS/DTjBF9fis4DDk+L8dyXJCwiWk3SVNtrz66sCyRNARYEjqGZtPX+uhHFeCDpSGB+mhZjgF2AGbb/o15U3ZAkLCJaTdJRNLfdn1CK3gtMtP3BelHVI2lV4IPAjsAlwNFZrLnbJF1te53ZlcWclyQsIlqtrJX4cWCzUjQFOLTLayaWSUrfAXwHeIBmrNx/2T61Zlz9JmkFYFXbv5X0AmA+2w/WjqvfJF0B7Gj7lrK9MnBK1lyd+5KERUSrSdoK+ENH53+ahaS1gd2At9LMEfZD21dIejlwke0VqgbYR5I+RDNn2tK2VykthIfZ3qpyaH1X/o0cDUynSchXAHazfW7VwDogSVhEtJqkY4GNgX/StIJdAPy+rJvYKZLOp5mc9ZTBSamkXWwfXyey/pN0FbAh8Efb65WyabZfUzWwSkqL8epl88YutxT3U6aoiIhWs/0BgNLaswPwPeDldOz/v9IFecdwiVaXErDicdtPSAJA0nxAp1olJE0G/mr777Yfl7QusD3wZ0n7Z7LWuW9C7QAiIuYmSe8rayWeArwR+C7NJJ2dYnsGzRxpC9SOZZw4X9J/AS+QtDVwMs2M8V1yOPAEgKTXA18BjqNZxumIinF1RrojI6LVJN0D3AIcBpxr+7a6EdUj6Tjg1cBpNGtqAmD7G9WCqkTSBGB34E0046DOAo50h34p9t4BKel7wD9s71+2r7K9bsXwOqFTzfER0T22XyRpTZqJKL9UBmDfaHuXyqHVcEt5TAAWqxxLVbafBn5QHl01UdJ8ZcWArWhuVBiQ/KAPUskR0WqSFgdeQXPH14rAEjTzhnWO7QNqxzBeSNoGOIjmezEfTWuYbS9eNbD++hFNt+w9wKM0N64g6ZU0XZIxl6U7MiJaTdJU4PflcYHt2yuH1HeSvmX7k5J+yRCDz21vWyGsqiTdDGwHTOtSF+RgkjYClgF+Y/vhUrYasKjtK6oG1wFJwiIiWk7SBrYvl/SGoV63fX6/Y6pN0rnAVqVbMqKKJGEREdE5ZXqGg4DzgWfmxOriTQpRT8aERUR0RLkp4cvAGsBCA+W2V64WVD1fAh6iqYdM2xFVJAmLiOiOo4H9gG8CW9AsYdTV+SJfbnut2kFEt6U7MiJaSdIhjDADuu29+hjOuCDpctsb9C7PM1BWO7Z+k3Qw8Fvbv6kdS3RXV/8Cioj2uwy4nKa7aX3gpvJYl+52Pz1eJim9SdKekt4JLFo7qEo+Cpwp6VFJD0h6UNIDtYOKbklLWES0mqSLgc3KhJRImh+YYnujupH1XxmMfj2wJM2g9MWBr9m+uGZcEV2VMWER0XZL0SQbA4sRL1rKOkPS8WWFgE1sX0ozIH23ymFVJ2kpYFVmvUnhgnoRRdckCYuItvsKcGWZF0o0yxftXzWi/ttA0suBD5b1I9X7ou1/Dn1Ye0n6D2BvYDngKmAj4CJgy4phRcekOzIiWk/Sy4DXls0/2v57zXj6TdJeNGOgVgbuYNYkzF2cokLSNGAycLHtdSW9Cvgf29tVDi06JAPzI6LVJAl4I7CO7V8AC0jasHJYfWX7O7ZfDRxle2XbK/U8OpeAFY/ZfgxA0oK2bwBWrxxTdEySsIhou0OBjYGdy/aDwPfqhVOP7Y/WjmEcuV3SksDPgbMl/QL4c9WIonPSHRkRrSbpCtvrS7rS9nql7Grb69SOLcaHsqbmEsCZtp+oHU90RwbmR0TbPSlpImXiVkkvBrJoc8dJWrpnc1r5mVaJ6Kt0R0ZE230H+BnwEklfAn4P/E/dkGIcuAL4B/Anmkl8/wHcJukKSZ1bQSDqSHdkRLReufNtK5q7An9n+/rKIVUhaSPgEODVNKsGTAQetr141cAqkPQD4BTbZ5XtNwHb06yv+W3brx3p+Ig5IUlYRLRemZRzeXqGYNi+ol5EdUi6DHg3cDIwCXg/sJrtz1UNrILe9TN7yqbaXlvSVbbXrRRadEjGhEVEq0k6CNgVuIWZY35MRyfltH2zpIm2ZwBHS7oS6FwSBtwp6f8BJ5XtnYC7yvjBjBmMvkgSFhFt9y5gldz1BsAjkhYArpJ0MHAn3R0b/B5gP5opKgxcWMom0nxnIua6dEdGRKtJ+inwUdt3146lNkkrAHfRjAf7FM20DIfavrlqYBEdlSQsIlpN0iTgF8A1wOMD5ba3rRZURATpjoyI9jsW+CrNXFAZ6xMR40ZawiKi1SRdanty7Thi/CiD7/ey/c3asUS3JQmLiFaT9A2absjTmLU7snNTVMRMki6x3amF3GP8SRIWEa0m6dwhim27c1NUSFoN+AywArPOmdbFuvgmMD/wY+DhgfIk59FPScIiIjpC0tXAYcDlwIyBctuXVwuqkiTnMR4kCYuIVpO071Dltg/sdyy1SbrcdtZFjBgnujpJX0R0x8M9jxnAvwMr1gyool9K+pikZSQtPfCoHVQNkl4q6YeSfl2215C0e+24olvSEhYRnSJpQeAs25vXjqXfJN06RLFtr9z3YCorydfRwH/bXkfSfMCVg9eTjJibMk9YRHTNwsBytYOowfZKtWMYR15k+yeSPgdg+ylJM2Z3UMSclCQsIlpN0jRmLtw9EXgxcFC9iPpP0pa2z5G03VCv2z613zGNAw9LeiHluyFpI+D+uiFF1yQJi4i226bn+VPAXbafqhVMJW8AzgHeNsRrBrqYhH2aZu64VSRdSJOc71A3pOiajAmLiFaTdLztXWZXFt1TxoGtDgi40faTlUOKjsndkRHRdmv2bpRfvJmmoeMkfRxY1Pa1tq8BFpX0sdpxRbckCYuIVpL0OUkPAmtLeqA8HgTuAn5RObyo70O27xvYsP0v4EP1wokuSndkRLSapC/b/lztOGJ8KTdsrO3yS7As6j3V9pojHxkx52RgfkS03emSFrH9sKT3AesD37b959qB1SBpE5rJanvXjjyuWkD1nAn8WNLhZfvDpSyib9ISFhGtJmkqsA6wNnAMcCTwLttvqBlXDZKOB1YBrmLm2pG2vVe1oCqRNAHYA3hjKTobONJ25gqLvkkSFhGtJukK2+uXNSTvsP3DgbLasfWbpOuBNZz/+JG0CPDYQNJVuiMXtP1I3ciiSzIwPyLa7sEyK/ouwK9KC8j8lWOq5RrgZbWDGCd+B7ygZ/sFwG8rxRIdlTFhEdF2OwHvAT5o+++SXgF8rXJMtbwIuE7SJcDjA4W2t60XUjUL2X5oYMP2Q5IWrhlQdE+SsIhotZJ4nQhMlrQNcElHB6ID7F87gHHkYUnr274CQNIGwKOVY4qOyZiwiGg1Se+iafk6j2Zm9NcBn7F9Ss24oi5Jk4GTgL/RfC9eBuxk+/KqgUWnJAmLiFaTdDWwte27y/aLgd/aXqduZP1XFqk+BHg1sADNguYP2168amCVSJqfZtkiyLJFUUG6IyOi7SYMJGDFvXT3pqTvAu8GTgYmAe8HVqsaUSUlAfso8PpSdJ6kw5OIRT+lJSwiWk3S12jmCPtRKdoJmGb7P+tFVYeky2xPkjTV9tql7Erb69WOrd8kHUlzl+yxpWgXYIbt/6gXVXRNWsIiotVsf0bSdsBmpegI2z+rGVNFj0haALhK0sHAnXS3VXDyoC7pc0rXdUTfJAmLiFaTtBJwhu1Ty/YLJK1o+7a6kVWxC804sD2BTwHLA9tXjaieGZJWsX0LgKSVmbmKQERfpDsyIlpN0mXAJrafKNsLABfanlw3sqhJ0lbA0cB0mrsjVwB2s31u1cCiU9ISFhFtN99AAgZg+4mSiHVOmSftIJqEYz6a5MNdvDvS9u8krcqsd0c+PtIxEXNaV8cCRER3/EPSMzPCS3o7cE/FeGr6FvAB4IW2F7e9WBcTMABJOwIL2J4KbAv8SFLn1hONutIdGRGtJmkV4ETg5aXodmCXgbFAXSLpXGAr20/XjqW2gTtEJW1G0zr4dWBf26+tHFp0SJKwiOgESYtCs0Zg7VhqKbPEHwScz6xrR36jWlCVDEzNIenLNFOW/F9Xp+uIejImLCI6ocvJV48vAQ8BC9HMmN9ld0g6HNga+KqkBckQneiztIRFRHSEpGtsr1U7jvFA0sLAm2lawW6StAzwGtu/qRxadEiSsIiIjigTtP42iUbE+JAkLCJarcyWP9j9NC0gdw/xWmtJehBYBHgCGFgjsZNTVESMB0nCIqLVJP0K2BgYmIRzc+ByYCXgQNvHVwotIjouA/Mjou3mA15t+y4ASS8FjgNeC1wAdCoJK3Omvb5snmf79JrxjBdlqoqdbX+8dizRHUnCIqLtlh9IwIq7S9k/JT053EFtJOkrwGSaedMA9pa0qe3PVQyrGknrAe8BdgRuBU6tG1F0TZKwiGi78ySdDpxctrcvZYsA91WLqo63AOsOTNYq6VjgSqAzSZik1YCdy+Me4Mc0Q3O2qBpYdFLGhEVEq0kSTeK1aSm6EPipO/ifn6SpwOa2/1m2l6bpkly7bmT9I+lpYAqwu+2bS9l02yvXjSy6KC1hEdFqJdk6pTy67svAlWX5ItGMDfts3ZD6bjvg3cC5ks4ETqKpi4i+S0tYRLRamaLiq8BLaH7Zig5Py1AmJZ1cNi+x/fea8dRSuqPfTtMtuSXNzRo/yxxq0U9JwiKi1STdDLzN9vW1Y6lN0gk060ZOsX1D7XjGC0lL0QzO38n2VrXjie5IEhYRrSbpQtubzn7P9pO0BfC68liFZlD+Bba/XTWwiI5KEhYRrSbp28DLgJ8Djw+U2+7kdASSJtJ0R24BfAR41Par6kYV0U0ZmB8Rbbc48Ajwpp4y08E5oST9jmbZooto7hCc3LWlmyLGkyRhEdFqtnerHcM4MhXYAFiLZv3M+yRdZPvRumFFdFO6IyOilST9p+2DJR1C0/I1C9t7VQhrXJC0GLArsA/wMtsL1o0oopvSEhYRbTVwN+RlVaMYRyTtSTMofwPgNuAomm7JiKggLWERER0haR+apOty20/Vjiei65KERUQrSfolQ3RDDrC9bR/DiYh4lnRHRkRbfb12ABERI0lLWEREREQFaQmLiFaS9BPb75I0jVm7JQfWjly7UmjVSPoEcILtf9WOJSKShEVEe+1dfm5TNYrx5aXApZKuoLkz8iynOySimnRHRkQnSFqcnj88bf+zYjjVSBLN6gG7AZOAnwA/tH1L1cAiOmhC7QAiIuYmSR+W9Hea2eIvL4/Ozh1WWr7+Xh5PAUsBp0g6uGpgER2UlrCIaDVJNwEb276ndiy1SdobeD9wD3Ak8HPbT0qaANxke5WqAUZ0TMaERUTb3UKzgHfA0sB2tv/cW2j7aUkZOxfRZ2kJi4hWk7QecDTwR+DxgfKurh0paX1gM5o7Ri+0fUXlkCI6Ky1hEdF2hwPnANOApyvHUpWkLwDvAk4tRUdLOtn2FyuGFdFZaQmLiFaTdKXt9WrHMR5IuhFYx/ZjZfsFwFW2V68bWUQ35e7IiGi7X0vaQ9IykpYeeNQOqpK/AQv1bC8I3FEplojOS0tYRLSapFuHKLbtlfseTGWSfg5MBs6mGRO2NXAJcDt0d5xcRC1JwiIiOkLSB0Z63fax/YolIpKERUTLSdoRONP2g5I+D6wPHGT7ysqhVSFpAeBVNC1hN9p+onJIEZ2VMWER0XZfKAnYZsAbgR8Ch1WOqQpJb6GZN+07wHeBmyX9e92oIrorSVhEtN2M8vOtwBG2fwUsUDGemr4BbGF7c9tvALYAvlk5pojOShIWEW13h6TDgZ2AMyQtSHf/73vQ9s0929OBB2sFE9F1GRMWEa0maWHgzcA02zdJWgZ4je3fVA6t7yR9H1gB+AnNmLAdgb8AvwWwferwR0fEnJYkLCKiIyQdPcLLtv3BvgUTEUnCIiIiImrI2pERER0haSFgd2BNembOTwtYRB1dHZwaER0laTNJ36sdRyXHAy8D/g04H1iODMyPqCZJWES0nqT1JH1N0m3AQcANlUOq5ZW2vwA8XGbHfyvw2soxRXRWuiMjopUkrQbsXB73AD+mGQe7RdXA6nqy/LxP0lrA34GXVIwnotOShEVEW90ATAG2GZgbS9Kn6oZU3RGSlgK+AJwGLArsWzekiO7K3ZER0UqS3gG8G9gUOBM4CTjS9ko144qIGJAkLCJaTdIiwNtpuiW3BI4DftbRyVoXBLYHVqSnJ8T2gbViiuiyJGER0RmlK25HYCfbW9WOp98knQncD1zOzDU1sf2/1YKK6LAkYRERHSHpGttr1Y4jIhqZoiIiojv+IOk1tYOIiEZawiIiWk7SNJoFu+cDVgWmA48Dolkzcu2K4UV0VpKwiIiWk7TCSK/b/nO/YomImZKERURERFSQMWERERERFSQJi4iIiKggSVhEREdJ2kzS92rHEdFVWTsyIqJDJK0HvIdm0tpbgVPrRhTRXUnCIiJaTtJqNMs27QzcA/yY5sasLaoGFtFxuTsyIqLlJD0NTAF2t31zKZtue+W6kUV0W8aERUS033bAncC5kn4gaSuaiVojoqK0hEVEdISkRYC303RLbgkcB/zM9m+qBhbRUUnCIiI6SNJSNIPzd7K9Ve14IrooSVhEREREBRkTFhEREVFBkrCIiIiICpKERURERFSQJCwiIiKigiRhERERERX8/7/WBJXAbR+YAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_heatmap(texts, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r9NudDYNxC46"
   },
   "source": [
    "## 2️⃣ 🤗 Transformers\n",
    "\n",
    "Alternatively, you can use the models straight from Hugging Face Transformers. This just requires a few extra steps. Just make sure to [install the Transformers library](https://huggingface.co/docs/transformers/installation) first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -U transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example using [DeCLUTR-small](https://huggingface.co/johngiorgi/declutr-small):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6a4Ymv39xC47"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModel, AutoTokenizer\n",
    "\n",
    "# Load the model\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"johngiorgi/declutr-small\")\n",
    "model = AutoModel.from_pretrained(\"johngiorgi/declutr-small\")\n",
    "model = model.to(device)\n",
    "\n",
    "# Prepare some text to embed\n",
    "text = [\n",
    "    \"A smiling costumed woman is holding an umbrella.\",\n",
    "    \"A happy woman in a fairy costume holds an umbrella.\",\n",
    "    \"A soccer game with multiple males playing.\",\n",
    "    \"Some men are playing a sport.\",\n",
    "]\n",
    "inputs = tokenizer(text, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "# Put the tensors on the GPU, if available\n",
    "for name, tensor in inputs.items():\n",
    "    inputs[name] = tensor.to(device)\n",
    "\n",
    "# Embed the text\n",
    "with torch.no_grad():\n",
    "    sequence_output = model(**inputs)[0]\n",
    "\n",
    "# Mean pool the token-level embeddings to get sentence-level embeddings\n",
    "embeddings = torch.sum(\n",
    "    sequence_output * inputs[\"attention_mask\"].unsqueeze(-1), dim=1\n",
    ") / torch.clamp(torch.sum(inputs[\"attention_mask\"], dim=1, keepdims=True), min=1e-9)\n",
    "embeddings = embeddings.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute the same cosine similarity as before as a sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.835675835609436\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "# Compute a semantic similarity via the cosine distance\n",
    "semantic_sim = 1 - cosine(embeddings[0], embeddings[1])\n",
    "print(semantic_sim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fSVlxjYexC4k"
   },
   "source": [
    "## 3️⃣ From this repo\n",
    "\n",
    "To use the model directly from this repo, import `Encoder` and pass it some text (it accepts both strings and lists of strings). Just remember to [install the DeCLUTR library](https://github.com/JohnGiorgi/DeCLUTR) first:\n",
    "\n",
    "> Note, you might have to reset the runtime (or kernel) before installing DeCLUTR and running the following cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U \"git+https://github.com/JohnGiorgi/DeCLUTR.git\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example using [DeCLUTR-small](https://huggingface.co/johngiorgi/declutr-small):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q8pG_xeZxC4m"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from declutr import Encoder\n",
    "\n",
    "# This can be a path on disk to a model you have trained yourself OR\n",
    "# the name of one of our pretrained models.\n",
    "pretrained_model_or_path = \"declutr-small\"\n",
    "\n",
    "# Prepare some text to embed\n",
    "text = [\n",
    "    \"A smiling costumed woman is holding an umbrella.\",\n",
    "    \"A happy woman in a fairy costume holds an umbrella.\",\n",
    "    \"A soccer game with multiple males playing.\",\n",
    "    \"Some men are playing a sport.\",\n",
    "]\n",
    "\n",
    "# Embed the text\n",
    "cuda_device = torch.cuda.current_device() if torch.cuda.is_available() else -1\n",
    "encoder = Encoder(pretrained_model_or_path, cuda_device=cuda_device)\n",
    "embeddings = encoder(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute the same cosine similarity as before as a sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.835675835609436\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "# Compute a semantic similarity via the cosine distance\n",
    "semantic_sim = 1 - cosine(embeddings[0], embeddings[1])\n",
    "print(semantic_sim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PAEXMEQ2xC41"
   },
   "source": [
    "See the list of available `PRETRAINED_MODELS` in [declutr/encoder.py](https://github.com/JohnGiorgi/DeCLUTR/blob/master/declutr/encoder.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "2imLEDWDxC42"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['declutr-small', 'declutr-base']\n"
     ]
    }
   ],
   "source": [
    "from declutr.encoder import PRETRAINED_MODELS ; print(list(PRETRAINED_MODELS.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QvpLSGecxC5C"
   },
   "source": [
    "## 4️⃣ Bulk embed a file\n",
    "\n",
    "First, lets save our running example to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SG4kpaJGxC5C"
   },
   "outputs": [],
   "source": [
    "text = [\n",
    "    \"A smiling costumed woman is holding an umbrella.\",\n",
    "    \"A happy woman in a fairy costume holds an umbrella.\",\n",
    "    \"A soccer game with multiple males playing.\",\n",
    "    \"Some men are playing a sport.\",\n",
    "]\n",
    "text = \"\\n\".join(text)\n",
    "\n",
    "!echo -e \"$text\" > \"input.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mOG-sN2SxC5H"
   },
   "source": [
    "We then need a pretrained model to embed the text with. Following our running example, lets use [DeCLUTR-small](https://huggingface.co/johngiorgi/declutr-small)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "0VoqRh7WxC5J"
   },
   "outputs": [],
   "source": [
    "from allennlp.common.file_utils import cached_path\n",
    "from declutr.encoder import PRETRAINED_MODELS\n",
    "\n",
    "# Download the model OR retrieve its filepath if it has already been downloaded & cached.\n",
    "declutr_small_cached_path = cached_path(PRETRAINED_MODELS[\"declutr-small\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eBQfez2axC5O"
   },
   "source": [
    "To embed all text in a given file with a trained model, run the following command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0wNYDQ7-xC5P"
   },
   "outputs": [],
   "source": [
    "# When embedding text with a pretrained model, we do NOT want to sample spans.\n",
    "# We can turn off span sampling by setting the num_anchors attribute to None.\n",
    "overrides = \"{'dataset_reader.num_anchors': null}\"\n",
    "\n",
    "!allennlp predict $declutr_small_cached_path \"input.txt\" \\\n",
    "    --output-file \"embeddings.jsonl\" \\\n",
    "    --batch-size 32 \\\n",
    "    --cuda-device $cuda_device \\\n",
    "    --use-dataset-reader \\\n",
    "    --overrides \"$overrides\" \\\n",
    "    --include-package \"declutr\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0u23mh0UxC5W"
   },
   "source": [
    "As a sanity check, lets load the embeddings and make sure their cosine similarity is as expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "NGYcpU3DxC5X"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(\"embeddings.jsonl\", \"r\") as f:\n",
    "    embeddings = []\n",
    "    for line in f:\n",
    "        embeddings.append(json.loads(line)[\"embeddings\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "OrsptLBmxC5d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8356757721781577\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "\n",
    "semantic_sim = 1 - cosine(embeddings[0], embeddings[1])\n",
    "print(semantic_sim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0mswpdqCxC5i"
   },
   "source": [
    "## ♻️ Conclusion\n",
    "\n",
    "That's it! In this notebook, we covered four ways to embed text with a pretrained model. Please see [our paper](https://arxiv.org/abs/2006.03659) and [repo](https://github.com/JohnGiorgi/DeCLUTR) for more details, and don't hesitate to open an issue if you have any trouble!"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "embedding.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "seq2rel",
   "language": "python",
   "name": "seq2rel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}