notebooks/evaluating.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "evaluating.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LY9LO9FnSPIa"
      },
      "source": [
        "# Evaluating a model\n",
        "\n",
        "This notebook will walk you through evaluating a [DeCLUTR](https://github.com/JohnGiorgi/DeCLUTR) model with [SentEval](https://github.com/facebookresearch/SentEval)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbZ4o1HHSM5t"
      },
      "source": [
        "## 🔧 Install the prerequisites\n",
        "\n",
        "Clone to repository locally so we have access to the evaluation scripts. Then install DeCLUTR"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fdyoe-EPSKLN"
      },
      "source": [
        "%%bash\n",
        "git clone https://github.com/JohnGiorgi/DeCLUTR.git\n",
        "cd DeCLUTR\n",
        "pip install -e .\n",
        "cd ../"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5o1owrdbWDl9"
      },
      "source": [
        "Next, we have to clone the SentEval benchmark locally (this will take a few minutes)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_9Mg77kOREs7"
      },
      "source": [
        "%%bash\n",
        "# Clone our fork which has several bug fixes merged\n",
        "git clone https://github.com/JohnGiorgi/SentEval.git\n",
        "cd SentEval/data/downstream/\n",
        "./get_transfer_data.bash\n",
        "cd ../../../"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m05mAP5wWU-f"
      },
      "source": [
        "Lastly, we need a model to evaluate. We will download `DeCLUTR-small`:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Nd8oYGpUn5k"
      },
      "source": [
        "!wget https://github.com/JohnGiorgi/DeCLUTR/releases/download/v0.1.0rc1/declutr-small.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ae7g5puWn8X"
      },
      "source": [
        "## 📋 Evaluating the model\n",
        "\n",
        "Finally, use our provided script to evaluate the model on SentEval.\n",
        "\n",
        "> Note, the script will evaluate on all 28 SentEval tasks. This can take 7 hours or more on a GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zLWGGQSJUwW1"
      },
      "source": [
        "!python DeCLUTR/scripts/run_senteval.py allennlp \"SentEval\" \"declutr-small.tar.gz\" \\\n",
        " --output-filepath \"senteval_results.json\" \\\n",
        " --cuda-device 0  \\\n",
        " --include-package \"declutr\" \\\n",
        " --verbose"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MLRyWa1IXepJ"
      },
      "source": [
        "We also provide commands for evaluating other popular sentence encoders. For a list of commands, run:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GWdKsXc_W6TC"
      },
      "source": [
        "!python DeCLUTR/scripts/run_senteval.py --help"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OCNuiVzLXozu"
      },
      "source": [
        "For help with a specific command, e.g. `transformers`, run:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XEbSmYIVWNoB"
      },
      "source": [
        "!python DeCLUTR/scripts/run_senteval.py transformers --help"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AZyXiAmHX3RF"
      },
      "source": [
        "Notice that evaluate other popular models, like [Sentence Transformers](https://www.sbert.net/)! Just make sure to install it first:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "samad0pSbg4N"
      },
      "source": [
        "!pip install sentence-transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V1kN71RBXtwB"
      },
      "source": [
        "!python DeCLUTR/scripts/run_senteval.py sentence-transformers \"SentEval\" \"roberta-base-nli-mean-tokens\" \\\n",
        " --output-filepath \"senteval_results.json\" \\\n",
        " --cuda-device 0  \\\n",
        " --verbose"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}