01. Transformer Models/TransformerModels.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TransformerModels.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**INITIALIZATION:**\n",
        "- I use these three lines of code on top of my each notebooks because it will help to prevent any problems while reloading the same project. And the third line of code helps to make visualization within the notebook."
      ],
      "metadata": {
        "id": "s_rod7u5Z3Bk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "7-c8RRG5XCK7"
      },
      "outputs": [],
      "source": [
        "#@ INITIALIZATION: \n",
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LIBRARIES AND DEPENDENCIES:**\n",
        "- I have downloaded all the libraries and dependencies required for the project in one particular cell."
      ],
      "metadata": {
        "id": "n9I8VhyfaBr7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ INSTALLING DEPENDENCIES: UNCOMMENT BELOW: \n",
        "# !pip install transformers[sentencepiece]"
      ],
      "metadata": {
        "id": "6xlT0WUMZ-62"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@ DOWNLOADING LIBRARIES AND DEPENDENCIES:\n",
        "import transformers\n",
        "from transformers import pipeline"
      ],
      "metadata": {
        "id": "NOL1UW_vaWtG"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Natural Language Processing**\n",
        "- **NLP** is a field of linguistics and machine learning focused on understanding everything related to human language. The aim of **NLP** tasks is not only to understand single words individually, but to be able to understand the context of those words. "
      ],
      "metadata": {
        "id": "CECqmYGOawxR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**SENTIMENT ANALYSIS:**"
      ],
      "metadata": {
        "id": "RkznUFUOcY3F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF SENTIMENT ANALYSIS PIPELINE:\n",
        "classifier = pipeline(\"sentiment-analysis\")                                 # Initializing Classifier Object. \n",
        "classifier(\"I've started the HuggingFace course which fascinates me.\")      # Inspecting Sentiment. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ibYIi6HBatZn",
        "outputId": "467f1699-c91c-4bc6-d47d-e8dcb1805f3d"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'label': 'POSITIVE', 'score': 0.9997233748435974}]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF SENTIMENT ANALYSIS PIPELINE: MULTIPLE:\n",
        "classifier([\"I've started the HuggingFace course which fascinates me.\",\n",
        "            \"I will no longer read it's documentation.\",\n",
        "            \"I think the course is awesome!\"])                              # Inspecting Sentiment. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pYpdpAjXc4dH",
        "outputId": "2f3ffbfe-456c-4914-8a2b-e72c84ace700"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'label': 'POSITIVE', 'score': 0.9997233748435974},\n",
              " {'label': 'NEGATIVE', 'score': 0.9994686245918274},\n",
              " {'label': 'POSITIVE', 'score': 0.9998465776443481}]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Pipelines**\n",
        "- The three main steps involved when we pass some text to a `pipeline` are: \n",
        "    - The text is preprocessed into a format the model can understand.\n",
        "    - The preprocessed inputs are passed to the model. \n",
        "    - The predictions of the model are post-processed, so we can make sense of them. "
      ],
      "metadata": {
        "id": "bp4Yx3MffEw6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**ZERO-SHOT CLASSIFICATION:**\n",
        "- The `zero-shot-classification` pipeline is very powerful, as it allows us to specify which labels to use for the classification, so we don't have to rely on the labels of the pretrained model. This `pipeline` is called `zero-shot` because we don't need to fine-tune the model on our data to use it. It can directly return probability scores for any list of labels we want. "
      ],
      "metadata": {
        "id": "natPtwAFgD9T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF ZERO SHOT CLASSIFICATION PIPELINE: \n",
        "classifier = pipeline(\"zero-shot-classification\")                           # Initializing Classifier Object. \n",
        "classifier(\"This is a course about Transformers library\", \n",
        "           candidate_labels=[\"education\", \"health\", \"programming\"])         # Inspecting Classification. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wt8ilH9VeW1A",
        "outputId": "cd15fd0e-b9ec-4d68-bd96-64d6bfa6ca14"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to facebook/bart-large-mnli (https://huggingface.co/facebook/bart-large-mnli)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['education', 'programming', 'health'],\n",
              " 'scores': [0.690150260925293, 0.19084124267101288, 0.11900852620601654],\n",
              " 'sequence': 'This is a course about Transformers library'}"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TEXT GENERATION:**\n",
        "- The main idea here is that when we provide a prompt and the model will auto-complete it by generating the remaining text. "
      ],
      "metadata": {
        "id": "HSCQbBQdylz6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: \n",
        "generator = pipeline(\"text-generation\")                                 # Initializing Generator Object. \n",
        "generator(\"In this course, you will learn to\")                          # Inspecting Generated Text. "
      ],
      "metadata": {
        "id": "GS1r-kAQirUE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8945cadb-dcfc-44c2-fb05-3d4a8768d3f9"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to gpt2 (https://huggingface.co/gpt2)\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'generated_text': 'In this course, you will learn to create functional web applications with AngularJS. One part of this course will be working with web components in real time, as well as web technologies in a way so that you can integrate them seamlessly. So be sure'}]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: \n",
        "generator(\"I like python because\", num_return_sequences=2, max_length=15)     # Inspecting Generated Sequences. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YeJPE8k5zjPL",
        "outputId": "7be625bb-9c1d-4258-83ca-a28f4832ea10"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'generated_text': 'I like python because of all that it makes my day.'},\n",
              " {'generated_text': 'I like python because of its amazing support for object oriented, modular code.'}]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: DISTILGPT2:\n",
        "generator = pipeline(\"text-generation\", model=\"distilgpt2\")                   # Initializing Generator Object. \n",
        "generator(\"I want to be a programmer so that\", \n",
        "          num_return_sequences=2, max_length=30)                              # Inspecting Generated Sequences. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i8tULAYiz_1g",
        "outputId": "682c579f-cbc4-4868-e46d-381b058715a1"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'generated_text': 'I want to be a programmer so that I can enjoy the world.'},\n",
              " {'generated_text': 'I want to be a programmer so that we can do the same thing. If you want to start making things, you really need to get creative,'}]"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**MASK FILLING:**\n",
        "- The idea of this task is to fill in the blanks in a given text. "
      ],
      "metadata": {
        "id": "J0zsBIMFCwDw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF MASK FILLING PIPELINE:\n",
        "unmasker = pipeline(\"fill-mask\")                                        # Initilizing Mask Filling Object. \n",
        "unmasker(\"This course will teach you all about <mask> models.\", \n",
        "         top_k=2)                                                       # Inspecting Mask Token."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4c1UCi9C2kI6",
        "outputId": "f5bd0254-94a7-4b03-b0d5-a61eb914e57f"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to distilroberta-base (https://huggingface.co/distilroberta-base)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'score': 0.196198508143425,\n",
              "  'sequence': 'This course will teach you all about mathematical models.',\n",
              "  'token': 30412,\n",
              "  'token_str': ' mathematical'},\n",
              " {'score': 0.040527332574129105,\n",
              "  'sequence': 'This course will teach you all about computational models.',\n",
              "  'token': 38163,\n",
              "  'token_str': ' computational'}]"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF MASK FILLING PIPELINE:\n",
        "unmasker = pipeline(\"fill-mask\", model=\"bert-base-cased\")               # Initilizing Mask Filling Object. \n",
        "unmasker(\"This course will teach you all about [MASK] models.\", \n",
        "         top_k=2)                                                       # Inspecting Mask Token."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kjvAPF2zHCgt",
        "outputId": "b33e49ac-a579-4831-e83d-66f62665fcbe"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'score': 0.2596317529678345,\n",
              "  'sequence': 'This course will teach you all about role models.',\n",
              "  'token': 1648,\n",
              "  'token_str': 'role'},\n",
              " {'score': 0.0942726731300354,\n",
              "  'sequence': 'This course will teach you all about the models.',\n",
              "  'token': 1103,\n",
              "  'token_str': 'the'}]"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**NAMED ENTITY RECOGNITION:**\n",
        "- **NER** is a task where the model has to find which parts of the input text correspond to entities such as persons, locations, or organizations. "
      ],
      "metadata": {
        "id": "T1IuXZA-Fdrd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF NER PIPELINE:\n",
        "ner = pipeline(\"ner\", grouped_entities=True)                           # Initializing NER Object. \n",
        "ner(\"I am Thinam Tamang and I am from Kathmandu, Nepal.\")              # Inspecting Entities. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xkHKDYCPFEZT",
        "outputId": "5af30e26-791a-4575-b7ca-525a481e9300"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to dbmdz/bert-large-cased-finetuned-conll03-english (https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english)\n",
            "/usr/local/lib/python3.7/dist-packages/transformers/pipelines/token_classification.py:136: UserWarning: `grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy=\"AggregationStrategy.SIMPLE\"` instead.\n",
            "  f'`grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy=\"{aggregation_strategy}\"` instead.'\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'end': 18,\n",
              "  'entity_group': 'PER',\n",
              "  'score': 0.9967239,\n",
              "  'start': 5,\n",
              "  'word': 'Thinam Tamang'},\n",
              " {'end': 42,\n",
              "  'entity_group': 'LOC',\n",
              "  'score': 0.9990222,\n",
              "  'start': 33,\n",
              "  'word': 'Kathmandu'},\n",
              " {'end': 49,\n",
              "  'entity_group': 'LOC',\n",
              "  'score': 0.9997008,\n",
              "  'start': 44,\n",
              "  'word': 'Nepal'}]"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**QUESTION ANSWERING:**\n",
        "- The `question-answering` pipeline answers questions using information from a given context. "
      ],
      "metadata": {
        "id": "G85OQxxHIBQp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF QUESTION ANSWERING PIPELINE: \n",
        "question_answerer = pipeline(\"question-answering\")                          # Initialization. \n",
        "question_answerer(\n",
        "    question=\"What is my name?\",\n",
        "    context=\"I am Thinam from Nepal.\")                                      # Inspecting Answer. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F4q1loPYH3pB",
        "outputId": "c16effd2-731a-4a2b-d1a7-d4c8ca51a8b1"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to distilbert-base-cased-distilled-squad (https://huggingface.co/distilbert-base-cased-distilled-squad)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'answer': 'Thinam', 'end': 11, 'score': 0.8733119964599609, 'start': 5}"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**SUMMARIZATION:**\n",
        "- Summarization is the task of reducing a text into a shorter text while keeping all or most of the important aspects referenced in the text. "
      ],
      "metadata": {
        "id": "uxANYti2Jzen"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF SUMMARIZATION PIPELINE:\n",
        "summarizer = pipeline(\"summarization\")                                                  # Initializing Summarizer Object. \n",
        "summarizer(\n",
        "    \"\"\"\n",
        "    America has changed dramatically during recent years. Not only has the number of \n",
        "    graduates in traditional engineering disciplines such as mechanical, civil, \n",
        "    electrical, chemical, and aeronautical engineering declined, but in most of \n",
        "    the premier American universities engineering curricula now concentrate on \n",
        "    and encourage largely the study of engineering science. As a result, there \n",
        "    are declining offerings in engineering subjects dealing with infrastructure, \n",
        "    the environment, and related issues, and greater concentration on high \n",
        "    technology subjects, largely supporting increasingly complex scientific \n",
        "    developments. While the latter is important, it should not be at the expense \n",
        "    of more traditional engineering.\n",
        "\n",
        "    Rapidly developing economies such as China and India, as well as other \n",
        "    industrial countries in Europe and Asia, continue to encourage and advance \n",
        "    the teaching of engineering. Both China and India, respectively, graduate \n",
        "    six and eight times as many traditional engineers as does the United States. \n",
        "    Other industrial countries at minimum maintain their output, while America \n",
        "    suffers an increasingly serious decline in the number of engineering graduates \n",
        "    and a lack of well-educated engineers.\n",
        "\"\"\"\n",
        ")                                                                                         # Inspecting Summarized Text. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2-jGKeWkJeVa",
        "outputId": "eb0d8461-e28c-444c-cc2b-604c68a2bfa2"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "No model was supplied, defaulted to sshleifer/distilbart-cnn-12-6 (https://huggingface.co/sshleifer/distilbart-cnn-12-6)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'summary_text': ' America has changed dramatically during recent years . The number of engineering graduates in the U.S. has declined in traditional engineering disciplines such as mechanical, civil,    electrical, chemical, and aeronautical engineering . Rapidly developing economies such as China and India continue to encourage and advance the teaching of engineering .'}]"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TRANSLATION:**"
      ],
      "metadata": {
        "id": "JukLFMN6LGRq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@ IMPLEMENTATION OF TRANSLATION PIPELINE: \n",
        "translator = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-fr-en\")        # Initializing Translator Object. \n",
        "translator(\"Ce cours est produit par Hugging Face.\")                            # Inspecting Translation. "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aEcD0rquKsMx",
        "outputId": "03b449f7-424a-4b5b-d834-526e9006ad9b"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'translation_text': 'This course is produced by Hugging Face.'}]"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    }
  ]
}