explore_data_and_model.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## High-level overview\n",
    "\n",
    "We can all agree researching products, platforms or services is painful and time taking.\n",
    "\n",
    "I often find myself researching multiple sources (R.I.P chrome tabs) to see if something is reliable, worthy, and reasonable to buy and still feel like I am lost. Information is scattered, hard to find, and hard to sort through (multiple mentions of different products, mixed reviews) and get a clear summary.\n",
    "\n",
    "Moreover, Misleading ads, SEO spam (best 5 product articles with the least research), and fake reviews (Affiliated links, sponsored content) can make the whole experience frustrating.\n",
    "\n",
    "Reddit is one place where people leave honest reviews and discourage any fake ones. With over 52 million daily active users, Reddit provides a platform for people from all walks of life to share their experiences and opinions.\n",
    "\n",
    "### The goal of this project is to help users find:\n",
    "1) if a product or service is worth it via sentiment analysis. This helps with queries such as \"Is regal unlimited subscription worth it?\"\n",
    "2) product mentions amongst posts and comments via named entity recognition (NER). This helps with queries such as \"Best 4K TV to buy\".\n",
    "\n",
    "Please see the set_up_notebook.ipynb that addresses first problem - sentiment analysis."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook focuses on solving the second problem - Named Entity Recognition\n",
    "\n",
    "Hugging Face's Transformers provides State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX.\n",
    "Training an NLP model from scratch takes hundreds of hours.\n",
    "Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce compute costs, carbon footprint, and save the time and resources required to train a model from scratch.\n",
    "\n",
    "\n",
    "https://huggingface.co/models"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we can get a pre-trained model for our task (NER), these pre-trained models are not trained on the product entities. We need to get a dataset that has product labels and fine tune the pre-trained model on product enntities to be able to predict product mentions from reddit.\n",
    "\n",
    "Let's load libraries!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading libraries\n",
    "\n",
    "import os\n",
    "from datasets import load_dataset, concatenate_datasets, load_metric\n",
    "from transformers import AutoTokenizer, DataCollatorForTokenClassification, AutoModelForTokenClassification, TrainingArguments, Trainer, create_optimizer, AutoModel\n",
    "from transformers import EarlyStoppingCallback\n",
    "import evaluate\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "#import tensorflow as tf\n",
    "#from transformers.keras_callbacks import KerasMetricCallback\n",
    "#from transformers.keras_callbacks import PushToHubCallback\n",
    "from transformers import pipeline\n",
    "import wandb\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the dataset that has product labels from HugginFace Datasets library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset wnut_17 (C:\\Users\\avanjavakam\\.cache\\huggingface\\datasets\\wnut_17\\wnut_17\\1.0.0\\077c7f08b8dbc800692e8c9186cdf3606d5849ab0e7be662e6135bb10eba54f9)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fa67e18721a48359394c5bfb48b0eb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'tokens', 'ner_tags'],\n",
       "        num_rows: 3394\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'tokens', 'ner_tags'],\n",
       "        num_rows: 1009\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['id', 'tokens', 'ner_tags'],\n",
       "        num_rows: 1287\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wnut = load_dataset(\"wnut_17\")\n",
    "wnut"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of Input Data\n",
    "\n",
    "WNUT 17: Emerging and Rare entity recognition (https://huggingface.co/datasets/wnut_17)\n",
    "This dataset was chosen among many available for the following reasons:\n",
    "1) Reddit has many previously-unseen entities in the context of emerging discussions which is what this dataset focuses on\n",
    "2) Size of the dataset is small but good enough to meet the project goals\n",
    "3) Faster training (relative to other large datasets) and GPU is not required\n",
    "4) It is trained on noisy user-generated text (similar to what's needed for this project)\n",
    "\n",
    "train dataset has 3394 rows, validation dataset has 1009 rows and test dataset has 1287 rows."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see pre-defined entities of the data. Product entities are of the most interest for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'O',\n",
       " 1: 'B-corporation',\n",
       " 2: 'I-corporation',\n",
       " 3: 'B-creative-work',\n",
       " 4: 'I-creative-work',\n",
       " 5: 'B-group',\n",
       " 6: 'I-group',\n",
       " 7: 'B-location',\n",
       " 8: 'I-location',\n",
       " 9: 'B-person',\n",
       " 10: 'I-person',\n",
       " 11: 'B-product',\n",
       " 12: 'I-product'}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_list = wnut[\"train\"].features[f\"ner_tags\"].feature.names\n",
    "id2tag = {id: tag for id, tag in enumerate(label_list)}\n",
    "id2tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'O',\n",
       " 1: 'B-corporation',\n",
       " 2: 'I-corporation',\n",
       " 3: 'B-creative-work',\n",
       " 4: 'I-creative-work',\n",
       " 5: 'B-group',\n",
       " 6: 'I-group',\n",
       " 7: 'B-location',\n",
       " 8: 'I-location',\n",
       " 9: 'B-person',\n",
       " 10: 'I-person',\n",
       " 11: 'B-product',\n",
       " 12: 'I-product'}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_list = wnut[\"test\"].features[f\"ner_tags\"].feature.names\n",
    "id2tag = {id: tag for id, tag in enumerate(label_list)}\n",
    "id2tag"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train data set is not large but transformers require a large data set.\n",
    "To address this, train and validation sets are combined. Test data set will be untouched for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'tokens', 'ner_tags'],\n",
       "    num_rows: 4403\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge train & validation sets\n",
    "train_dataset = concatenate_datasets([wnut[\"train\"],wnut[\"validation\"]])\n",
    "train_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '2',\n",
       " 'tokens': ['Pxleyes',\n",
       "  'Top',\n",
       "  '50',\n",
       "  'Photography',\n",
       "  'Contest',\n",
       "  'Pictures',\n",
       "  'of',\n",
       "  'August',\n",
       "  '2010',\n",
       "  '...',\n",
       "  'http://bit.ly/bgCyZ0',\n",
       "  '#photography'],\n",
       " 'ner_tags': [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pxleyes', 'Top', '50', 'Photography', 'Contest', 'Pictures', 'of', 'August', '2010', '...', 'http://bit.ly/bgCyZ0', '#photography']\n",
      "['B-corporation', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "# looking at an example\n",
    "ith_example=2\n",
    "\n",
    "print(wnut[\"train\"][ith_example]['tokens'])\n",
    "print([id2tag[label] for label in train_dataset[ith_example]['ner_tags']])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pxleyes is tagged as a corporation and the rest are 'O' which means they do not belong to any entity"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Strategy for solving the problem\n",
    "\n",
    "1) First, a pre-trained model is needed from the Hugging Face.\n",
    "    Encoder\tmodels such as ALBERT, BERT, DistilBERT, ELECTRA, RoBERTa can be used for NER (also called token-classification).\n",
    "\n",
    "    These models are really large. I chose DistilBERT for it's faster training while meeting the functionality.\n",
    "\n",
    "    Distilbert aims to optimize the training by reducing the size of BERT and increase the speed of BERT — all while trying to retain as much performance as possible. Specifically, Distilbert is 40% smaller than the original BERT-base model, is 60% faster than it, and retains 97% of its functionality.\n",
    "2) This model needs to be fine tuned with wnut dataset in order to predict product entities.\n",
    "3) Evaluate how the model performs with test dataset based on accuracy (why this metric was chosen will be explained below)\n",
    "4) Test with a few example comments from reddit\n",
    "5) Push the model to Hugging Face hub so it can be used by the streamlit app for inference"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion of the expected solution\n",
    "\n",
    "For example, a user searches \"Best 4K TV to buy\" on the app, the model needs to identify the products that are mentioned in all the comments and posts on Reddit.\n",
    "\n",
    "For example,\n",
    "text=\"\"\"Sony X90K is one of the best TVs ever if you are in a room that has a lot of sunlight. But if you game a lot, nothing can beat LG C2 OLED. If you are tight on budget, go with TCL 6 series\"\"\"\n",
    "\n",
    "The model should tag Sony X90K, LG C2 OLED and TCL 6 Series as products."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics with justification\n",
    "\n",
    "Accuracy can be priortized over F1 score for this classification task considering the context and consequences.\n",
    "\n",
    "1) There are no siginificant consequences for false positives or false negatives.\n",
    "\n",
    "2) F1 score improvement requires a very balanced dataset (no of entities and non-entities being roughly equal) which is not the case for this project where text will have extremely high non-entities and only a handful of entities. F1 score is influenced by recall, which can be significantly impacted by imbalanced datasets.\n",
    "\n",
    "3) Accuracy is label-level metric that measures the overall correctness of the predicted labels compared to the true labels. It provides a straightforward and intuitive measure of how well the model predicts the entities as a whole. As the primary focus of this NER task is on overall correct labeling, accuracy is a suitable metric.\n",
    "\n",
    "4) It is hard to find a good decision threshold (usually 0.5) to provide a better trade-off between precision and recall for this specific problem as an entity and a non-entity should both be predicted correctly.\n",
    "\n",
    "5) Optimizing for F1 score requires annotated reddit dataset (fine tuning specifically for products) which I couldn't find and may involve manual labelling which is extremely expensive in terms of time and cost.\n",
    "\n",
    "6)  Accurately recognizing non-entities (tokens that do not correspond to any entity) is equally important as identifying the correct entity spans. Accuracy considers both entity and non-entity predictions, providing an overall measure of the model's performance in recognizing both types\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA\n",
    "\n",
    "Get the pre-trained distilbert model and it's tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"distilbert-base-uncased\"\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "exploring how tokenizer behaves with an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_input = tokenizer(wnut[\"train\"][2][\"tokens\"], is_split_into_words=True)\n",
    "tokens = tokenizer.convert_ids_to_tokens(tokenized_input[\"input_ids\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how the original input looks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pxleyes', 'Top', '50', 'Photography', 'Contest', 'Pictures', 'of', 'August', '2010', '...', 'http://bit.ly/bgCyZ0', '#photography']\n"
     ]
    }
   ],
   "source": [
    "#input\n",
    "print(wnut[\"train\"][2][\"tokens\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distilbert's tokenizer splits them into sub tokens. Special tokens CLS and SEP are added too. This results in a mismatch between the inputs and the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'p', '##xley', '##es', 'top', '50', 'photography', 'contest', 'pictures', 'of', 'august', '2010', '.', '.', '.', 'http', ':', '/', '/', 'bit', '.', 'l', '##y', '/', 'b', '##gc', '##y', '##z', '##0', '#', 'photography', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "#tokenized\n",
    "print(tokens)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "Re-alignment of tokens and labels involves:\n",
    "1. Each token is mapped to its tag\n",
    "2. assign label -100 to unnecessary tokens. PyTorch ignores -100 value during loss calculation.\n",
    "3. Sub tokens such as [‘p’, ‘##xley’, ‘##es’] should become [1,-100,-100] so we can re align.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_align_labels(examples):\n",
    "    tokenized_inputs = tokenizer(examples[\"tokens\"], truncation=True, is_split_into_words=True)\n",
    "\n",
    "    labels = []\n",
    "    for i, label in enumerate(examples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.\n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids:  # Set the special tokens to -100.\n",
    "            if word_idx is None:\n",
    "                label_ids.append(-100)\n",
    "            elif word_idx != previous_word_idx:  # Only label the first token of a given word.\n",
    "                label_ids.append(label[word_idx])\n",
    "            else:\n",
    "                label_ids.append(-100)\n",
    "            previous_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function tokenize_and_align_labels at 0x000002488D9641F0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "38a8e5b7317e466eb8bc7c6a394213e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c28490bb64b4bbe868a6221e85990f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b757ebbef7c4aa381562f5022d9ec76",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "816a48c8ae614fe0b9d616c2e36521e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# applying the custom token function\n",
    "tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)\n",
    "tokenized_train_dataset = train_dataset.map(tokenize_and_align_labels, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tokens</th>\n",
       "      <th>ner_labels</th>\n",
       "      <th>ner_tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[CLS]</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>p</td>\n",
       "      <td>1</td>\n",
       "      <td>B-corporation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>##xley</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>##es</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>top</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>photography</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>contest</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>pictures</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>of</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>august</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2010</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>.</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>.</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>.</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>http</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>:</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>/</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>/</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>bit</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>.</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>l</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>##y</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>/</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>b</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>##gc</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>##y</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>##z</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>##0</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>#</td>\n",
       "      <td>0</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>photography</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>[SEP]</td>\n",
       "      <td>-100</td>\n",
       "      <td>ignore</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         tokens  ner_labels       ner_tags\n",
       "0         [CLS]        -100         ignore\n",
       "1             p           1  B-corporation\n",
       "2        ##xley        -100         ignore\n",
       "3          ##es        -100         ignore\n",
       "4           top           0              O\n",
       "5            50           0              O\n",
       "6   photography           0              O\n",
       "7       contest           0              O\n",
       "8      pictures           0              O\n",
       "9            of           0              O\n",
       "10       august           0              O\n",
       "11         2010           0              O\n",
       "12            .           0              O\n",
       "13            .        -100         ignore\n",
       "14            .        -100         ignore\n",
       "15         http           0              O\n",
       "16            :        -100         ignore\n",
       "17            /        -100         ignore\n",
       "18            /        -100         ignore\n",
       "19          bit        -100         ignore\n",
       "20            .        -100         ignore\n",
       "21            l        -100         ignore\n",
       "22          ##y        -100         ignore\n",
       "23            /        -100         ignore\n",
       "24            b        -100         ignore\n",
       "25         ##gc        -100         ignore\n",
       "26          ##y        -100         ignore\n",
       "27          ##z        -100         ignore\n",
       "28          ##0        -100         ignore\n",
       "29            #           0              O\n",
       "30  photography        -100         ignore\n",
       "31        [SEP]        -100         ignore"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2tag[-100]='ignore'\n",
    "exml=tokenized_train_dataset[2]\n",
    "\n",
    "pd.DataFrame({'tokens':tokenizer.convert_ids_to_tokens(exml[\"input_ids\"]), 'ner_labels':exml['labels'], 'ner_tags': [id2tag[label] for label in exml['labels']] })"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "Initial check:\n",
    "Check how a simple baseline model performs. This model can just tag every token with the most frequent entity throughout the data which is O.\n",
    "\n",
    "This resulted in ~59% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5888494815191806"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.dummy import DummyClassifier\n",
    "\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "dummy_clf.fit(pd.Series(tokenized_train_dataset['input_ids']).explode(), pd.Series(tokenized_train_dataset['labels']).explode().astype(str))\n",
    "dummy_clf.score(pd.Series(tokenized_train_dataset['input_ids']).explode(), pd.Series(tokenized_train_dataset['labels']).explode().astype(str))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The baseline classifier becomes less naive if we tag each token with the most frequent label of the sentence it belongs.\n",
    "This resulted in 72% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7197897448947134"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exploded_values=pd.Series(tokenized_train_dataset['labels']).explode()\n",
    "exploded_values=pd.DataFrame(exploded_values,columns=['B'])\n",
    "\n",
    "most_frequent_elem_by_doc=pd.Series(tokenized_train_dataset['labels']).apply(lambda x:  max(set(x), key=x.count))\n",
    "most_frequent_elem_by_doc=pd.DataFrame(most_frequent_elem_by_doc,columns=list('A'))\n",
    "\n",
    "df_most_freq_token=exploded_values.merge(most_frequent_elem_by_doc, how='right', left_index=True, right_index=True)\n",
    "\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "dummy_clf.fit(pd.Series(tokenized_train_dataset['input_ids']).explode(), df_most_freq_token['A'])\n",
    "dummy_clf.score(pd.Series(tokenized_train_dataset['input_ids']).explode(), df_most_freq_token['A'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using DistilBERT for Named Entity Recognition\n",
    "\n",
    "labels should be padded the exact same way as the inputs so that they stay the same size, using -100 as a value so that the corresponding predictions are ignored in the loss computation.\n",
    "\n",
    "DataCollatorForTokenClassification helps with this padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "# Data Collator\n",
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)\n",
    "\n",
    "# Get model\n",
    "model = AutoModelForTokenClassification.from_pretrained(model_name, num_labels=len(label_list))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a quick evaluation before we fine tune the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'corporation': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1},\n",
       " 'overall_precision': 1.0,\n",
       " 'overall_recall': 1.0,\n",
       " 'overall_f1': 1.0,\n",
       " 'overall_accuracy': 1.0}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metric_seqeval = load_metric(\"seqeval\")\n",
    "example = wnut[\"train\"][2]\n",
    "\n",
    "labels = [label_list[i] for i in example[\"ner_tags\"]]\n",
    "metric_seqeval.compute(predictions=[labels], references=[labels])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the precision, recall, and F1 score for each separate entity, as well as overall.\n",
    "To get metrics on the validation set during training, we need to define the function that'll calculate the metric for us. This is very well-documented in their official docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(p):\n",
    "    predictions, labels = p\n",
    "    predictions = np.argmax(predictions, axis=2)\n",
    "\n",
    "    # Remove ignored index (special tokens)\n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "\n",
    "    results = metric_seqeval.compute(predictions=true_predictions, references=true_labels)\n",
    "    flattened_results = {\n",
    "        \"precision\": results[\"overall_precision\"],\n",
    "        \"recall\": results[\"overall_recall\"],\n",
    "        \"f1\": results[\"overall_f1\"],\n",
    "        \"accuracy\": results[\"overall_accuracy\"],\n",
    "    }\n",
    "    # adding entity level metric as well\n",
    "    #for k in results.keys():\n",
    "    #    if k not in flattened_results.keys():\n",
    "    #        flattened_results[k+\"_f1\"]=results[k][\"f1\"]\n",
    "    return flattened_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17f8105b8b97476db3a372e3f2c5adb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1380 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a DistilBertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3bdb5caccdf249f1a2bbbdf85dfcb862",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.8291836977005005, 'eval_precision': 0.014814814814814815, 'eval_recall': 0.005560704355885079, 'eval_f1': 0.008086253369272238, 'eval_accuracy': 0.910820401008935, 'eval_runtime': 101.8258, 'eval_samples_per_second': 12.639, 'eval_steps_per_second': 0.206, 'epoch': 0.22}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa22bef3bd094ea2a6476cfc3efac4ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.458038866519928, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 111.7945, 'eval_samples_per_second': 11.512, 'eval_steps_per_second': 0.188, 'epoch': 0.43}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b6f6f9629764a5b84332b7a48053d96",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3688420057296753, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 108.5973, 'eval_samples_per_second': 11.851, 'eval_steps_per_second': 0.193, 'epoch': 0.65}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c17578fa0721480b87e009d251cc8340",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.31115037202835083, 'eval_precision': 0.39375, 'eval_recall': 0.11677479147358666, 'eval_f1': 0.18012866333095068, 'eval_accuracy': 0.9325809071865248, 'eval_runtime': 105.4344, 'eval_samples_per_second': 12.207, 'eval_steps_per_second': 0.199, 'epoch': 0.87}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd220f5e766f469c9c7cd388c50e940e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2867625951766968, 'eval_precision': 0.4480769230769231, 'eval_recall': 0.2159406858202039, 'eval_f1': 0.29143214509068166, 'eval_accuracy': 0.9368133042623231, 'eval_runtime': 107.3207, 'eval_samples_per_second': 11.992, 'eval_steps_per_second': 0.196, 'epoch': 1.09}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f00296f85d24370a20f55531a3916b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2620603144168854, 'eval_precision': 0.41881443298969073, 'eval_recall': 0.30120481927710846, 'eval_f1': 0.35040431266846367, 'eval_accuracy': 0.9385661151725022, 'eval_runtime': 108.3989, 'eval_samples_per_second': 11.873, 'eval_steps_per_second': 0.194, 'epoch': 1.3}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83c947105a4445099439994f651c70c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2517760097980499, 'eval_precision': 0.5279672578444747, 'eval_recall': 0.3586654309545876, 'eval_f1': 0.4271523178807947, 'eval_accuracy': 0.9431832756188278, 'eval_runtime': 122.2843, 'eval_samples_per_second': 10.525, 'eval_steps_per_second': 0.172, 'epoch': 1.52}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8b264850c7aa4f10a4d7b961bed96b8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.24010969698429108, 'eval_precision': 0.4760845383759733, 'eval_recall': 0.39666357738646896, 'eval_f1': 0.43276036400404455, 'eval_accuracy': 0.9421144884784747, 'eval_runtime': 94.176, 'eval_samples_per_second': 13.666, 'eval_steps_per_second': 0.223, 'epoch': 1.74}\n",
      "{'loss': 0.5257, 'learning_rate': 2e-05, 'epoch': 1.81}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba1c989ee05647c2897048ea6741b790",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.24359002709388733, 'eval_precision': 0.5162846803377563, 'eval_recall': 0.39666357738646896, 'eval_f1': 0.44863731656184486, 'eval_accuracy': 0.9433542815612842, 'eval_runtime': 82.8193, 'eval_samples_per_second': 15.54, 'eval_steps_per_second': 0.254, 'epoch': 1.96}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d07847a4cd94dfc982b951ff899de3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2503388226032257, 'eval_precision': 0.5418275418275418, 'eval_recall': 0.3901760889712697, 'eval_f1': 0.45366379310344823, 'eval_accuracy': 0.9458766192125176, 'eval_runtime': 83.163, 'eval_samples_per_second': 15.476, 'eval_steps_per_second': 0.253, 'epoch': 2.17}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f6fdec7ba6443438a87179918a1449e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2353149950504303, 'eval_precision': 0.46853823814133594, 'eval_recall': 0.448563484708063, 'eval_f1': 0.45833333333333337, 'eval_accuracy': 0.9451498439570775, 'eval_runtime': 82.081, 'eval_samples_per_second': 15.68, 'eval_steps_per_second': 0.256, 'epoch': 2.39}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "865a49097b8340a18c903b9c3e7c25e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.24050550162792206, 'eval_precision': 0.5618279569892473, 'eval_recall': 0.3873957367933272, 'eval_f1': 0.4585847504114098, 'eval_accuracy': 0.9475866786370827, 'eval_runtime': 86.6375, 'eval_samples_per_second': 14.855, 'eval_steps_per_second': 0.242, 'epoch': 2.61}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33f0db7feeb94756b50f98f0daec67ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.23974569141864777, 'eval_precision': 0.5839112343966713, 'eval_recall': 0.3901760889712697, 'eval_f1': 0.46777777777777774, 'eval_accuracy': 0.9481424479500663, 'eval_runtime': 86.3845, 'eval_samples_per_second': 14.899, 'eval_steps_per_second': 0.243, 'epoch': 2.83}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a957fa2bf9547a1a8384bd2b3eb3eaf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.245252326130867, 'eval_precision': 0.5699873896595208, 'eval_recall': 0.41890639481000924, 'eval_f1': 0.48290598290598286, 'eval_accuracy': 0.948398956863751, 'eval_runtime': 83.9294, 'eval_samples_per_second': 15.334, 'eval_steps_per_second': 0.25, 'epoch': 3.04}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24666ec2d67e444ab311cf92ba037457",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2514230012893677, 'eval_precision': 0.5508474576271186, 'eval_recall': 0.42168674698795183, 'eval_f1': 0.4776902887139107, 'eval_accuracy': 0.9486982172630499, 'eval_runtime': 81.402, 'eval_samples_per_second': 15.81, 'eval_steps_per_second': 0.258, 'epoch': 3.26}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8f9328873f54443e8c47d5f32891ebe6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.26181262731552124, 'eval_precision': 0.5732814526588845, 'eval_recall': 0.40963855421686746, 'eval_f1': 0.47783783783783784, 'eval_accuracy': 0.9491257321191912, 'eval_runtime': 85.1954, 'eval_samples_per_second': 15.106, 'eval_steps_per_second': 0.246, 'epoch': 3.48}\n",
      "{'loss': 0.0843, 'learning_rate': 8.636363636363637e-06, 'epoch': 3.62}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0718f2af33d4cc08e35837413894f7a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.2565685510635376, 'eval_precision': 0.5918097754293263, 'eval_recall': 0.4151992585727525, 'eval_f1': 0.48801742919389973, 'eval_accuracy': 0.9493394895472618, 'eval_runtime': 92.7674, 'eval_samples_per_second': 13.873, 'eval_steps_per_second': 0.226, 'epoch': 3.7}\n",
      "{'train_runtime': 4502.8795, 'train_samples_per_second': 4.889, 'train_steps_per_second': 0.306, 'train_loss': 0.3002868703767365, 'epoch': 3.7}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=1020, training_loss=0.3002868703767365, metrics={'train_runtime': 4502.8795, 'train_samples_per_second': 4.889, 'train_steps_per_second': 0.306, 'train_loss': 0.3002868703767365, 'epoch': 3.7})"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='C:/Users/avanjavakam/OneDrive - Moulton Niguel Water/Documents/R_Home/distilbert_ner',\n",
    "    #report_to=\"wandb\",\n",
    "    #run_name = \"initial_run\"\n",
    "    num_train_epochs=5,\n",
    "    learning_rate=2e-5,\n",
    "    per_device_train_batch_size=16,   \n",
    "    per_device_eval_batch_size=64,\n",
    "    weight_decay=0.01,\n",
    "    warmup_steps=500, \n",
    "    eval_steps=60,\n",
    "    save_steps=60,\n",
    "    evaluation_strategy=\"steps\",\n",
    "    load_best_model_at_end=True\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_train_dataset,\n",
    "    eval_dataset=tokenized_wnut[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=data_collator,\n",
    "    compute_metrics=compute_metrics,\n",
    "    callbacks = [EarlyStoppingCallback(early_stopping_patience = 6)]\n",
    ")\n",
    "\n",
    "trainer.train()\n",
    "#wandb.finish()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test Set Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5389b390153e4753814254bac44e7a88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'corporation': {'precision': 0.21008403361344538,\n",
       "  'recall': 0.3787878787878788,\n",
       "  'f1': 0.27027027027027023,\n",
       "  'number': 66},\n",
       " 'creative-work': {'precision': 0.3392857142857143,\n",
       "  'recall': 0.13380281690140844,\n",
       "  'f1': 0.1919191919191919,\n",
       "  'number': 142},\n",
       " 'group': {'precision': 0.3194444444444444,\n",
       "  'recall': 0.1393939393939394,\n",
       "  'f1': 0.19409282700421943,\n",
       "  'number': 165},\n",
       " 'location': {'precision': 0.49382716049382713,\n",
       "  'recall': 0.5333333333333333,\n",
       "  'f1': 0.5128205128205128,\n",
       "  'number': 150},\n",
       " 'person': {'precision': 0.5914972273567468,\n",
       "  'recall': 0.745920745920746,\n",
       "  'f1': 0.6597938144329898,\n",
       "  'number': 429},\n",
       " 'product': {'precision': 0.20481927710843373,\n",
       "  'recall': 0.13385826771653545,\n",
       "  'f1': 0.16190476190476194,\n",
       "  'number': 127},\n",
       " 'overall_precision': 0.46853823814133594,\n",
       " 'overall_recall': 0.448563484708063,\n",
       " 'overall_f1': 0.45833333333333337,\n",
       " 'overall_accuracy': 0.9451498439570775}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions, labels, _ = trainer.predict(tokenized_wnut[\"test\"])\n",
    "predictions = np.argmax(predictions, axis=2)\n",
    "\n",
    "# Remove ignored index (special tokens)\n",
    "true_predictions = [\n",
    "    [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "    for prediction, label in zip(predictions, labels)\n",
    "]\n",
    "true_labels = [\n",
    "    [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "    for prediction, label in zip(predictions, labels)\n",
    "]\n",
    "\n",
    "results = metric_seqeval.compute(predictions=true_predictions, references=true_labels)\n",
    "results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tag_sentence(text:str):\n",
    "    # convert our text to a  tokenized sequence\n",
    "    inputs = tokenizer(text, truncation=True, return_tensors=\"pt\")#.to(\"cuda\")\n",
    "    # get outputs\n",
    "    outputs = model(**inputs)\n",
    "    # convert to probabilities with softmax\n",
    "    probs = outputs[0][0].softmax(1)\n",
    "    # get the tags with the highest probability\n",
    "    word_tags = [(tokenizer.decode(inputs['input_ids'][0][i].item()), id2tag[tagid.item()]) \n",
    "                  for i, tagid in enumerate (probs.argmax(axis=1))]\n",
    "\n",
    "    return pd.DataFrame(word_tags, columns=['word', 'tag'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "few examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         word            tag\n",
      "0       [CLS]              O\n",
      "1       apple  B-corporation\n",
      "2          un              O\n",
      "3        ##ve              O\n",
      "4       ##ils              O\n",
      "5         all              O\n",
      "6           -              O\n",
      "7         new              O\n",
      "8         mac      B-product\n",
      "9      ##book      I-product\n",
      "10        air      I-product\n",
      "11          ,              O\n",
      "12      super              O\n",
      "13  ##charged              O\n",
      "14         by              O\n",
      "15        the              O\n",
      "16        new              O\n",
      "17         m2      B-product\n",
      "18       chip      I-product\n",
      "19      [SEP]              O\n"
     ]
    }
   ],
   "source": [
    "text=\"\"\"Apple unveils all-new MacBook Air, supercharged by the new M2 chip\"\"\"\n",
    "\n",
    "print(tag_sentence(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git config --global user.email \"anudeepvanjavakam@gmail.com\"\n",
    "!git config --global user.name \"anudeepvanjavakam1\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ffb79efe6965420d9c5636bbef04c0da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "notebook_login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.push_to_hub(commit_message=\"Training complete\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "289d8fb01590443087d9f316399a03bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/1.11k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a08c1c749a1944bda30e1e2ee6856040",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/266M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = AutoModelForTokenClassification.from_pretrained(\"anudeepvanjavakam/distilbert_uncased_finetuned_wnut17\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8971bfa136e34ee188655838f53b2cdc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/320 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41f3efd82b26422786da9d6bd074b657",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "59ad775f2f884c04a2b7ad8e68e92f9f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f66b59d936c74ffeb4536a5a27ad9ba3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer = Trainer(model=model)\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"anudeepvanjavakam/distilbert_uncased_finetuned_wnut17\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set up Weights and Biases for tracking and monitoring model runs.\n",
    "#Later we can use it for hyperparameter tuning.\n",
    "# set the wandb project where this run will be logged\n",
    "os.environ[\"WANDB_PROJECT\"]=\"reddit_product_tagging\"\n",
    "\n",
    "# save your trained model checkpoint to wandb\n",
    "os.environ[\"WANDB_LOG_MODEL\"]=\"true\"\n",
    "\n",
    "# turn off watch to log faster\n",
    "os.environ[\"WANDB_WATCH\"]=\"false\"\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During hyperparameter search, the Trainer will run several trainings, so it needs to have the model defined via a function (so it can be reinitialized at each new run) instead of just having it passed. We just use the same function as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_init():\n",
    "    return AutoModelForTokenClassification.from_pretrained(model_name, num_labels=len(label_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    #output_dir='C:/Users/avanjavakam/OneDrive - Moulton Niguel Water/Documents/R_Home/distilbert_ner_wandb',\n",
    "    report_to=\"wandb\",\n",
    "    run_name = \"run_2_06_04_2023_11_44_pm\",\n",
    "    disable_tqdm=True,\n",
    "    #num_train_epochs=5,\n",
    "    #learning_rate=2e-5,\n",
    "    #per_device_train_batch_size=16,   \n",
    "    #per_device_eval_batch_size=64,\n",
    "    #weight_decay=0.01,\n",
    "    #warmup_steps=500, \n",
    "    eval_steps=500,\n",
    "    #save_steps=60,\n",
    "    evaluation_strategy=\"steps\",\n",
    "    load_best_model_at_end=True,\n",
    "    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model_init=model_init,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_train_dataset,\n",
    "    eval_dataset=tokenized_wnut[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=data_collator,\n",
    "    compute_metrics=compute_metrics\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Population-Based Training\n",
    "Population-based training uses guided hyperparameter search but does not need to restart training for new hyperparameter configurations. Instead of discarding bad-performing trials, we exploit good-performing runs by copying their network weights and hyperparameters and then explore new hyperparameter configurations while continuing to train.\n",
    "\n",
    "The basic idea behind the algorithm in layman's terms:\n",
    "Run the hyperparameter optimization process for some samples for a given time step (or iterations) T.\n",
    "\n",
    "After every T iterations, compare the runs and copy the weights of good-performing runs to the bad-performing runs and change their hyperparameter values to be close to the runs' values that performed well.\n",
    "\n",
    "Terminate the worst-performing runs.\n",
    "Although the algorithm's idea seems simple, there is a lot of complex optimization math that goes into building this from scratch. Tune provides a scalable and easy-to-use implementation of the SOTA PBT algorithm\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray import tune\n",
    "from ray.tune.schedulers import PopulationBasedTraining\n",
    "\n",
    "def get_scheduler():\n",
    "    #Creating the PBT scheduler\n",
    "    scheduler = PopulationBasedTraining(\n",
    "        mode = \"max\",\n",
    "        metric='eval_acc',\n",
    "        #perturbation_interval=2,\n",
    "        hyperparam_mutations={\n",
    "            \"weight_decay\": tune.choice([0.0, 0.3]),\n",
    "            \"learning_rate\": tune.choice([1e-5, 5e-5]),\n",
    "            \"per_device_train_batch_size\": tune.choice([16, 64]),\n",
    "            \"num_train_epochs\": tune.choice([2,5]),\n",
    "            \"warmup_steps\": tune.choice(range(0, 500))\n",
    "        }\n",
    "    )\n",
    "    return scheduler"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run only 8 trials, much less than Bayesian Optimization since instead of stopping bad trials, they copy from the good ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Create sweep with ID: ile8sxcl\n",
      "Sweep URL: https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: zggjdxl0 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 6.187091203956564e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 4\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 8\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_002115-zggjdxl0</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">gentle-sweep-1</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "115a700569184d6e88484dfb8212ac94",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">gentle-sweep-1</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_002115-zggjdxl0\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_002139-zggjdxl0</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">gentle-sweep-1</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 2044.8051, 'train_samples_per_second': 8.613, 'train_steps_per_second': 0.135, 'train_loss': 0.14894058393395465, 'epoch': 4.0}\n",
      "{'eval_loss': 0.2613465487957001, 'eval_precision': 0.5406976744186046, 'eval_recall': 0.4309545875810936, 'eval_f1': 0.47962867457452296, 'eval_accuracy': 0.9479286905219957, 'eval_runtime': 62.4535, 'eval_samples_per_second': 20.607, 'eval_steps_per_second': 2.578, 'epoch': 4.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁</td></tr><tr><td>train/global_step</td><td>▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94793</td></tr><tr><td>eval/f1</td><td>0.47963</td></tr><tr><td>eval/loss</td><td>0.26135</td></tr><tr><td>eval/precision</td><td>0.5407</td></tr><tr><td>eval/recall</td><td>0.43095</td></tr><tr><td>eval/runtime</td><td>62.4535</td></tr><tr><td>eval/samples_per_second</td><td>20.607</td></tr><tr><td>eval/steps_per_second</td><td>2.578</td></tr><tr><td>train/epoch</td><td>4.0</td></tr><tr><td>train/global_step</td><td>276</td></tr><tr><td>train/total_flos</td><td>268255832904750.0</td></tr><tr><td>train/train_loss</td><td>0.14894</td></tr><tr><td>train/train_runtime</td><td>2044.8051</td></tr><tr><td>train/train_samples_per_second</td><td>8.613</td></tr><tr><td>train/train_steps_per_second</td><td>0.135</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">gentle-sweep-1</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/zggjdxl0</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_002139-zggjdxl0\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: 3crqic57 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 6.749872973956807e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 5\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 16\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 40\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_005708-3crqic57</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">colorful-sweep-2</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79b1d0bec2364a56847cd441387d1206",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">colorful-sweep-2</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_005708-3crqic57\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_005729-3crqic57</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">colorful-sweep-2</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.15, 'learning_rate': 4.304266823972457e-05, 'epoch': 1.81}\n",
      "{'eval_loss': 0.245904341340065, 'eval_precision': 0.5497448979591837, 'eval_recall': 0.3994439295644115, 'eval_f1': 0.4626945786366076, 'eval_accuracy': 0.9480141934932239, 'eval_runtime': 64.192, 'eval_samples_per_second': 20.049, 'eval_steps_per_second': 2.508, 'epoch': 1.81}\n",
      "{'loss': 0.0341, 'learning_rate': 1.8586606739881063e-05, 'epoch': 3.62}\n",
      "{'eval_loss': 0.32771819829940796, 'eval_precision': 0.5997150997150997, 'eval_recall': 0.3901760889712697, 'eval_f1': 0.47276810780460415, 'eval_accuracy': 0.9482707024069087, 'eval_runtime': 61.776, 'eval_samples_per_second': 20.833, 'eval_steps_per_second': 2.606, 'epoch': 3.62}\n",
      "{'train_runtime': 2982.7351, 'train_samples_per_second': 7.381, 'train_steps_per_second': 0.463, 'train_loss': 0.07024156148882879, 'epoch': 5.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁█</td></tr><tr><td>eval/f1</td><td>▁█</td></tr><tr><td>eval/loss</td><td>▁█</td></tr><tr><td>eval/precision</td><td>▁█</td></tr><tr><td>eval/recall</td><td>█▁</td></tr><tr><td>eval/runtime</td><td>█▁</td></tr><tr><td>eval/samples_per_second</td><td>▁█</td></tr><tr><td>eval/steps_per_second</td><td>▁█</td></tr><tr><td>train/epoch</td><td>▁▁▅▅█</td></tr><tr><td>train/global_step</td><td>▁▁▅▅█</td></tr><tr><td>train/learning_rate</td><td>█▁</td></tr><tr><td>train/loss</td><td>█▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94827</td></tr><tr><td>eval/f1</td><td>0.47277</td></tr><tr><td>eval/loss</td><td>0.32772</td></tr><tr><td>eval/precision</td><td>0.59972</td></tr><tr><td>eval/recall</td><td>0.39018</td></tr><tr><td>eval/runtime</td><td>61.776</td></tr><tr><td>eval/samples_per_second</td><td>20.833</td></tr><tr><td>eval/steps_per_second</td><td>2.606</td></tr><tr><td>train/epoch</td><td>5.0</td></tr><tr><td>train/global_step</td><td>1380</td></tr><tr><td>train/learning_rate</td><td>2e-05</td></tr><tr><td>train/loss</td><td>0.0341</td></tr><tr><td>train/total_flos</td><td>292581540592470.0</td></tr><tr><td>train/train_loss</td><td>0.07024</td></tr><tr><td>train/train_runtime</td><td>2982.7351</td></tr><tr><td>train/train_samples_per_second</td><td>7.381</td></tr><tr><td>train/train_steps_per_second</td><td>0.463</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">colorful-sweep-2</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/3crqic57</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_005729-3crqic57\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: kgy00di9 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 5.6309788887288427e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 4\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 8\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 25\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_014752-kgy00di9</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">floral-sweep-3</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49868ff1908e484e8a634cb16201c204",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.011 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=0.108078…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">floral-sweep-3</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_014752-kgy00di9\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_014814-kgy00di9</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">floral-sweep-3</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.185, 'learning_rate': 4.35353358729308e-05, 'epoch': 0.91}\n",
      "{'eval_loss': 0.24270012974739075, 'eval_precision': 0.5070603337612324, 'eval_recall': 0.366079703429101, 'eval_f1': 0.4251883745963402, 'eval_accuracy': 0.9448933350433928, 'eval_runtime': 66.955, 'eval_samples_per_second': 19.222, 'eval_steps_per_second': 2.405, 'epoch': 0.91}\n",
      "{'loss': 0.0794, 'learning_rate': 3.0760882858573164e-05, 'epoch': 1.81}\n",
      "{'eval_loss': 0.2586621344089508, 'eval_precision': 0.6069868995633187, 'eval_recall': 0.386468952734013, 'eval_f1': 0.47225368063420153, 'eval_accuracy': 0.9482707024069087, 'eval_runtime': 64.5231, 'eval_samples_per_second': 19.946, 'eval_steps_per_second': 2.495, 'epoch': 1.81}\n",
      "{'loss': 0.0404, 'learning_rate': 1.798642984421554e-05, 'epoch': 2.72}\n",
      "{'eval_loss': 0.28830501437187195, 'eval_precision': 0.577023498694517, 'eval_recall': 0.40963855421686746, 'eval_f1': 0.47913279132791325, 'eval_accuracy': 0.9482279509212945, 'eval_runtime': 64.632, 'eval_samples_per_second': 19.913, 'eval_steps_per_second': 2.491, 'epoch': 2.72}\n",
      "{'loss': 0.02, 'learning_rate': 5.211976829857912e-06, 'epoch': 3.63}\n",
      "{'eval_loss': 0.321499228477478, 'eval_precision': 0.5828947368421052, 'eval_recall': 0.41056533827618164, 'eval_f1': 0.4817835780315388, 'eval_accuracy': 0.9486127142918216, 'eval_runtime': 62.421, 'eval_samples_per_second': 20.618, 'eval_steps_per_second': 2.579, 'epoch': 3.63}\n",
      "{'train_runtime': 2976.5202, 'train_samples_per_second': 5.917, 'train_steps_per_second': 0.74, 'train_loss': 0.07536158587668638, 'epoch': 4.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁▇▇█</td></tr><tr><td>eval/f1</td><td>▁▇██</td></tr><tr><td>eval/loss</td><td>▁▂▅█</td></tr><tr><td>eval/precision</td><td>▁█▆▆</td></tr><tr><td>eval/recall</td><td>▁▄██</td></tr><tr><td>eval/runtime</td><td>█▄▄▁</td></tr><tr><td>eval/samples_per_second</td><td>▁▅▄█</td></tr><tr><td>eval/steps_per_second</td><td>▁▅▄█</td></tr><tr><td>train/epoch</td><td>▁▁▃▃▅▅▇▇█</td></tr><tr><td>train/global_step</td><td>▁▁▃▃▅▅▇▇█</td></tr><tr><td>train/learning_rate</td><td>█▆▃▁</td></tr><tr><td>train/loss</td><td>█▄▂▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94861</td></tr><tr><td>eval/f1</td><td>0.48178</td></tr><tr><td>eval/loss</td><td>0.3215</td></tr><tr><td>eval/precision</td><td>0.58289</td></tr><tr><td>eval/recall</td><td>0.41057</td></tr><tr><td>eval/runtime</td><td>62.421</td></tr><tr><td>eval/samples_per_second</td><td>20.618</td></tr><tr><td>eval/steps_per_second</td><td>2.579</td></tr><tr><td>train/epoch</td><td>4.0</td></tr><tr><td>train/global_step</td><td>2204</td></tr><tr><td>train/learning_rate</td><td>1e-05</td></tr><tr><td>train/loss</td><td>0.02</td></tr><tr><td>train/total_flos</td><td>214620490744380.0</td></tr><tr><td>train/train_loss</td><td>0.07536</td></tr><tr><td>train/train_runtime</td><td>2976.5202</td></tr><tr><td>train/train_samples_per_second</td><td>5.917</td></tr><tr><td>train/train_steps_per_second</td><td>0.74</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">floral-sweep-3</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/kgy00di9</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_014814-kgy00di9\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: p7qe59dp with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 8.262120435766388e-06\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 5\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 14\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_023832-p7qe59dp</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">twilight-sweep-4</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2fa93e69ba24716a156f7e8f6ce83bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.011 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=0.108343…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">twilight-sweep-4</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_023832-p7qe59dp\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_023852-p7qe59dp</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">twilight-sweep-4</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 2571.1666, 'train_samples_per_second': 8.562, 'train_steps_per_second': 0.134, 'train_loss': 0.3223885356516078, 'epoch': 5.0}\n",
      "{'eval_loss': 0.2883630394935608, 'eval_precision': 0.40644171779141103, 'eval_recall': 0.24559777571825764, 'eval_f1': 0.3061813980358174, 'eval_accuracy': 0.9371980676328503, 'eval_runtime': 63.0104, 'eval_samples_per_second': 20.425, 'eval_steps_per_second': 2.555, 'epoch': 5.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁</td></tr><tr><td>train/global_step</td><td>▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.9372</td></tr><tr><td>eval/f1</td><td>0.30618</td></tr><tr><td>eval/loss</td><td>0.28836</td></tr><tr><td>eval/precision</td><td>0.40644</td></tr><tr><td>eval/recall</td><td>0.2456</td></tr><tr><td>eval/runtime</td><td>63.0104</td></tr><tr><td>eval/samples_per_second</td><td>20.425</td></tr><tr><td>eval/steps_per_second</td><td>2.555</td></tr><tr><td>train/epoch</td><td>5.0</td></tr><tr><td>train/global_step</td><td>345</td></tr><tr><td>train/total_flos</td><td>334327638245460.0</td></tr><tr><td>train/train_loss</td><td>0.32239</td></tr><tr><td>train/train_runtime</td><td>2571.1666</td></tr><tr><td>train/train_samples_per_second</td><td>8.562</td></tr><tr><td>train/train_steps_per_second</td><td>0.134</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">twilight-sweep-4</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/p7qe59dp</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_023852-p7qe59dp\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: runqcxqh with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 4.695388712117811e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 3\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 32\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 18\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_032304-runqcxqh</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">kind-sweep-5</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d663cd49362403692c9232f049afb95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">kind-sweep-5</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_032304-runqcxqh\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_032325-runqcxqh</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">kind-sweep-5</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 1564.0155, 'train_samples_per_second': 8.446, 'train_steps_per_second': 0.265, 'train_loss': 0.15146775176559668, 'epoch': 3.0}\n",
      "{'eval_loss': 0.26040154695510864, 'eval_precision': 0.5753968253968254, 'eval_recall': 0.4031510658016682, 'eval_f1': 0.47411444141689374, 'eval_accuracy': 0.9476294301226967, 'eval_runtime': 64.73, 'eval_samples_per_second': 19.883, 'eval_steps_per_second': 2.487, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁</td></tr><tr><td>train/global_step</td><td>▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94763</td></tr><tr><td>eval/f1</td><td>0.47411</td></tr><tr><td>eval/loss</td><td>0.2604</td></tr><tr><td>eval/precision</td><td>0.5754</td></tr><tr><td>eval/recall</td><td>0.40315</td></tr><tr><td>eval/runtime</td><td>64.73</td></tr><tr><td>eval/samples_per_second</td><td>19.883</td></tr><tr><td>eval/steps_per_second</td><td>2.487</td></tr><tr><td>train/epoch</td><td>3.0</td></tr><tr><td>train/global_step</td><td>414</td></tr><tr><td>train/total_flos</td><td>188982887544060.0</td></tr><tr><td>train/train_loss</td><td>0.15147</td></tr><tr><td>train/train_runtime</td><td>1564.0155</td></tr><tr><td>train/train_samples_per_second</td><td>8.446</td></tr><tr><td>train/train_steps_per_second</td><td>0.265</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">kind-sweep-5</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/runqcxqh</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_032325-runqcxqh\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: d6qthlc6 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 3.730120361942406e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 6\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 4\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 19\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_035052-d6qthlc6</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">zesty-sweep-6</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53bec61a354041ccb6f50f426eac5235",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">zesty-sweep-6</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_035052-d6qthlc6\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_035113-d6qthlc6</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">zesty-sweep-6</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.2391, 'learning_rate': 3.447792148050307e-05, 'epoch': 0.45}\n",
      "{'eval_loss': 0.2832615375518799, 'eval_precision': 0.6374045801526718, 'eval_recall': 0.30954587581093607, 'eval_f1': 0.41671865252651286, 'eval_accuracy': 0.9428412637339147, 'eval_runtime': 68.637, 'eval_samples_per_second': 18.751, 'eval_steps_per_second': 2.346, 'epoch': 0.45}\n",
      "{'loss': 0.1374, 'learning_rate': 3.165463934158208e-05, 'epoch': 0.91}\n",
      "{'eval_loss': 0.23936323821544647, 'eval_precision': 0.5104740904079382, 'eval_recall': 0.42910101946246526, 'eval_f1': 0.4662638469284995, 'eval_accuracy': 0.9454491043563764, 'eval_runtime': 60.063, 'eval_samples_per_second': 21.427, 'eval_steps_per_second': 2.681, 'epoch': 0.91}\n",
      "{'loss': 0.0881, 'learning_rate': 2.8831357202661102e-05, 'epoch': 1.36}\n",
      "{'eval_loss': 0.2819172739982605, 'eval_precision': 0.5716234652114598, 'eval_recall': 0.38832252085264135, 'eval_f1': 0.4624724061810155, 'eval_accuracy': 0.9472874182377837, 'eval_runtime': 70.51, 'eval_samples_per_second': 18.253, 'eval_steps_per_second': 2.283, 'epoch': 1.36}\n",
      "{'loss': 0.0811, 'learning_rate': 2.6008075063740118e-05, 'epoch': 1.82}\n",
      "{'eval_loss': 0.2832833230495453, 'eval_precision': 0.5882352941176471, 'eval_recall': 0.3614457831325301, 'eval_f1': 0.4477611940298508, 'eval_accuracy': 0.946346885554273, 'eval_runtime': 60.4831, 'eval_samples_per_second': 21.279, 'eval_steps_per_second': 2.662, 'epoch': 1.82}\n",
      "{'loss': 0.0497, 'learning_rate': 2.318479292481913e-05, 'epoch': 2.27}\n",
      "{'eval_loss': 0.3261525630950928, 'eval_precision': 0.5814285714285714, 'eval_recall': 0.37720111214087115, 'eval_f1': 0.45756042720629564, 'eval_accuracy': 0.9476721816083109, 'eval_runtime': 70.206, 'eval_samples_per_second': 18.332, 'eval_steps_per_second': 2.293, 'epoch': 2.27}\n",
      "{'loss': 0.0404, 'learning_rate': 2.0361510785898143e-05, 'epoch': 2.72}\n",
      "{'eval_loss': 0.3394909203052521, 'eval_precision': 0.5988620199146515, 'eval_recall': 0.3901760889712697, 'eval_f1': 0.4725028058361392, 'eval_accuracy': 0.9479714420076097, 'eval_runtime': 62.423, 'eval_samples_per_second': 20.617, 'eval_steps_per_second': 2.579, 'epoch': 2.72}\n",
      "{'loss': 0.0326, 'learning_rate': 1.753822864697716e-05, 'epoch': 3.18}\n",
      "{'eval_loss': 0.3297465145587921, 'eval_precision': 0.5712530712530712, 'eval_recall': 0.4309545875810936, 'eval_f1': 0.49128367670364503, 'eval_accuracy': 0.9492967380616476, 'eval_runtime': 61.217, 'eval_samples_per_second': 21.024, 'eval_steps_per_second': 2.63, 'epoch': 3.18}\n",
      "{'loss': 0.0165, 'learning_rate': 1.4714946508056175e-05, 'epoch': 3.63}\n",
      "{'eval_loss': 0.35687947273254395, 'eval_precision': 0.5659630606860159, 'eval_recall': 0.39759036144578314, 'eval_f1': 0.4670658682634731, 'eval_accuracy': 0.9481851994356804, 'eval_runtime': 62.239, 'eval_samples_per_second': 20.678, 'eval_steps_per_second': 2.587, 'epoch': 3.63}\n",
      "{'loss': 0.0171, 'learning_rate': 1.189166436913519e-05, 'epoch': 4.09}\n",
      "{'eval_loss': 0.3854111135005951, 'eval_precision': 0.5943396226415094, 'eval_recall': 0.4087117701575533, 'eval_f1': 0.4843492586490939, 'eval_accuracy': 0.9485699628062075, 'eval_runtime': 60.285, 'eval_samples_per_second': 21.349, 'eval_steps_per_second': 2.671, 'epoch': 4.09}\n",
      "{'loss': 0.0084, 'learning_rate': 9.068382230214204e-06, 'epoch': 4.54}\n",
      "{'eval_loss': 0.3695051372051239, 'eval_precision': 0.5640050697084917, 'eval_recall': 0.41241890639481, 'eval_f1': 0.47644539614561027, 'eval_accuracy': 0.9480141934932239, 'eval_runtime': 66.7115, 'eval_samples_per_second': 19.292, 'eval_steps_per_second': 2.413, 'epoch': 4.54}\n",
      "{'loss': 0.0126, 'learning_rate': 6.245100091293219e-06, 'epoch': 5.0}\n",
      "{'eval_loss': 0.37051981687545776, 'eval_precision': 0.5655940594059405, 'eval_recall': 0.4235403151065802, 'eval_f1': 0.4843667196608373, 'eval_accuracy': 0.9485699628062075, 'eval_runtime': 62.4236, 'eval_samples_per_second': 20.617, 'eval_steps_per_second': 2.579, 'epoch': 5.0}\n",
      "{'loss': 0.0047, 'learning_rate': 3.421817952372234e-06, 'epoch': 5.45}\n",
      "{'eval_loss': 0.398918092250824, 'eval_precision': 0.5721784776902887, 'eval_recall': 0.4040778498609824, 'eval_f1': 0.4736556219445953, 'eval_accuracy': 0.9484844598349793, 'eval_runtime': 64.0532, 'eval_samples_per_second': 20.093, 'eval_steps_per_second': 2.514, 'epoch': 5.45}\n",
      "{'loss': 0.0063, 'learning_rate': 5.985358134512488e-07, 'epoch': 5.9}\n",
      "{'eval_loss': 0.3886580765247345, 'eval_precision': 0.5620347394540943, 'eval_recall': 0.4198331788693234, 'eval_f1': 0.4806366047745358, 'eval_accuracy': 0.948398956863751, 'eval_runtime': 61.062, 'eval_samples_per_second': 21.077, 'eval_steps_per_second': 2.637, 'epoch': 5.9}\n",
      "{'train_runtime': 6339.8579, 'train_samples_per_second': 4.167, 'train_steps_per_second': 1.042, 'train_loss': 0.05568572189495342, 'epoch': 6.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁▄▆▅▆▇█▇▇▇▇▇▇</td></tr><tr><td>eval/f1</td><td>▁▆▅▄▅▆█▆▇▇▇▆▇</td></tr><tr><td>eval/loss</td><td>▃▁▃▃▅▅▅▆▇▇▇██</td></tr><tr><td>eval/precision</td><td>█▁▄▅▅▆▄▄▆▄▄▄▄</td></tr><tr><td>eval/recall</td><td>▁█▆▄▅▆█▆▇▇█▆▇</td></tr><tr><td>eval/runtime</td><td>▇▁█▁█▃▂▂▁▅▃▄▂</td></tr><tr><td>eval/samples_per_second</td><td>▂█▁█▁▆▇▆█▃▆▅▇</td></tr><tr><td>eval/steps_per_second</td><td>▂█▁█▁▆▇▆█▃▆▅▇</td></tr><tr><td>train/epoch</td><td>▁▁▂▂▂▂▃▃▃▃▄▄▄▄▅▅▆▆▆▆▇▇▇▇███</td></tr><tr><td>train/global_step</td><td>▁▁▂▂▂▂▃▃▃▃▄▄▄▄▅▅▆▆▆▆▇▇▇▇███</td></tr><tr><td>train/learning_rate</td><td>█▇▇▆▆▅▄▄▃▃▂▂▁</td></tr><tr><td>train/loss</td><td>█▅▃▃▂▂▂▁▁▁▁▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.9484</td></tr><tr><td>eval/f1</td><td>0.48064</td></tr><tr><td>eval/loss</td><td>0.38866</td></tr><tr><td>eval/precision</td><td>0.56203</td></tr><tr><td>eval/recall</td><td>0.41983</td></tr><tr><td>eval/runtime</td><td>61.062</td></tr><tr><td>eval/samples_per_second</td><td>21.077</td></tr><tr><td>eval/steps_per_second</td><td>2.637</td></tr><tr><td>train/epoch</td><td>6.0</td></tr><tr><td>train/global_step</td><td>6606</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.0063</td></tr><tr><td>train/total_flos</td><td>286614713103450.0</td></tr><tr><td>train/train_loss</td><td>0.05569</td></tr><tr><td>train/train_runtime</td><td>6339.8579</td></tr><tr><td>train/train_samples_per_second</td><td>4.167</td></tr><tr><td>train/train_steps_per_second</td><td>1.042</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">zesty-sweep-6</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/d6qthlc6</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_035113-d6qthlc6\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: krqmpvqg with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 6.708243353123604e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 2\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 21\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_053738-krqmpvqg</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">celestial-sweep-7</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20befc2471724021a66bc82224c75e0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.011 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=0.108272…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">celestial-sweep-7</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_053738-krqmpvqg\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_053759-krqmpvqg</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">celestial-sweep-7</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 1115.7473, 'train_samples_per_second': 7.892, 'train_steps_per_second': 0.124, 'train_loss': 0.2310990181522093, 'epoch': 2.0}\n",
      "{'eval_loss': 0.2490411102771759, 'eval_precision': 0.5442536327608983, 'eval_recall': 0.3818350324374421, 'eval_f1': 0.44880174291939, 'eval_accuracy': 0.945320849899534, 'eval_runtime': 65.359, 'eval_samples_per_second': 19.691, 'eval_steps_per_second': 2.463, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4adb11090df40ba98cd1b54dc7d4d32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='254.130 MB of 254.130 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0,…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁</td></tr><tr><td>train/global_step</td><td>▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94532</td></tr><tr><td>eval/f1</td><td>0.4488</td></tr><tr><td>eval/loss</td><td>0.24904</td></tr><tr><td>eval/precision</td><td>0.54425</td></tr><tr><td>eval/recall</td><td>0.38184</td></tr><tr><td>eval/runtime</td><td>65.359</td></tr><tr><td>eval/samples_per_second</td><td>19.691</td></tr><tr><td>eval/steps_per_second</td><td>2.463</td></tr><tr><td>train/epoch</td><td>2.0</td></tr><tr><td>train/global_step</td><td>138</td></tr><tr><td>train/total_flos</td><td>134024930102310.0</td></tr><tr><td>train/train_loss</td><td>0.2311</td></tr><tr><td>train/train_runtime</td><td>1115.7473</td></tr><tr><td>train/train_samples_per_second</td><td>7.892</td></tr><tr><td>train/train_steps_per_second</td><td>0.124</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">celestial-sweep-7</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/krqmpvqg</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_053759-krqmpvqg\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: 9k7iuegg with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 7.389668947799434e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tnum_train_epochs: 3\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tper_device_train_batch_size: 4\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tseed: 4\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_055802-9k7iuegg</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">pleasant-sweep-8</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set _wandb in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set assignments in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Trying to set metric in the hyperparameter search but there is no corresponding field in `TrainingArguments`.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f3a630c943e14025b63fbfd657e34ae5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.010 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=0.111453…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">pleasant-sweep-8</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_055802-9k7iuegg\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_055827-9k7iuegg</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">pleasant-sweep-8</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/ile8sxcl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'per_device_train_batch_size' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'num_train_epochs' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'seed' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.2209, 'learning_rate': 6.271039073775904e-05, 'epoch': 0.45}\n",
      "{'eval_loss': 0.2422882318496704, 'eval_precision': 0.5013333333333333, 'eval_recall': 0.3484708063021316, 'eval_f1': 0.41115363586659376, 'eval_accuracy': 0.9451498439570775, 'eval_runtime': 61.197, 'eval_samples_per_second': 21.03, 'eval_steps_per_second': 2.631, 'epoch': 0.45}\n",
      "{'loss': 0.1492, 'learning_rate': 5.152409199752375e-05, 'epoch': 0.91}\n",
      "{'eval_loss': 0.3045324981212616, 'eval_precision': 0.6195121951219512, 'eval_recall': 0.3531047265987025, 'eval_f1': 0.4498229043683589, 'eval_accuracy': 0.9448505835577786, 'eval_runtime': 71.46, 'eval_samples_per_second': 18.01, 'eval_steps_per_second': 2.253, 'epoch': 0.91}\n",
      "{'loss': 0.0875, 'learning_rate': 4.033779325728846e-05, 'epoch': 1.36}\n",
      "{'eval_loss': 0.32817161083221436, 'eval_precision': 0.6302816901408451, 'eval_recall': 0.33178869323447635, 'eval_f1': 0.4347298117789921, 'eval_accuracy': 0.9458766192125176, 'eval_runtime': 61.267, 'eval_samples_per_second': 21.006, 'eval_steps_per_second': 2.628, 'epoch': 1.36}\n",
      "{'loss': 0.0692, 'learning_rate': 2.915149451705317e-05, 'epoch': 1.82}\n",
      "{'eval_loss': 0.2856837213039398, 'eval_precision': 0.5457102672292545, 'eval_recall': 0.35959221501390176, 'eval_f1': 0.43351955307262563, 'eval_accuracy': 0.9457056132700611, 'eval_runtime': 63.874, 'eval_samples_per_second': 20.149, 'eval_steps_per_second': 2.521, 'epoch': 1.82}\n",
      "{'loss': 0.0512, 'learning_rate': 1.796519577681788e-05, 'epoch': 2.27}\n",
      "{'eval_loss': 0.340984970331192, 'eval_precision': 0.554945054945055, 'eval_recall': 0.3744207599629286, 'eval_f1': 0.44714997232982845, 'eval_accuracy': 0.9458766192125176, 'eval_runtime': 64.039, 'eval_samples_per_second': 20.097, 'eval_steps_per_second': 2.514, 'epoch': 2.27}\n",
      "{'loss': 0.0282, 'learning_rate': 6.778897036582586e-06, 'epoch': 2.72}\n",
      "{'eval_loss': 0.3519895374774933, 'eval_precision': 0.6043613707165109, 'eval_recall': 0.35959221501390176, 'eval_f1': 0.4509006391632771, 'eval_accuracy': 0.9469026548672567, 'eval_runtime': 63.4169, 'eval_samples_per_second': 20.294, 'eval_steps_per_second': 2.539, 'epoch': 2.72}\n",
      "{'train_runtime': 3104.1516, 'train_samples_per_second': 4.255, 'train_steps_per_second': 1.064, 'train_loss': 0.09467396662518937, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▂▁▅▄▅█</td></tr><tr><td>eval/f1</td><td>▁█▅▅▇█</td></tr><tr><td>eval/loss</td><td>▁▅▆▄▇█</td></tr><tr><td>eval/precision</td><td>▁▇█▃▄▇</td></tr><tr><td>eval/recall</td><td>▄▅▁▆█▆</td></tr><tr><td>eval/runtime</td><td>▁█▁▃▃▃</td></tr><tr><td>eval/samples_per_second</td><td>█▁█▆▆▆</td></tr><tr><td>eval/steps_per_second</td><td>█▁█▆▆▆</td></tr><tr><td>train/epoch</td><td>▁▁▂▂▃▃▅▅▆▆▇▇█</td></tr><tr><td>train/global_step</td><td>▁▁▂▂▃▃▅▅▆▆▇▇█</td></tr><tr><td>train/learning_rate</td><td>█▇▅▄▂▁</td></tr><tr><td>train/loss</td><td>█▅▃▂▂▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.9469</td></tr><tr><td>eval/f1</td><td>0.4509</td></tr><tr><td>eval/loss</td><td>0.35199</td></tr><tr><td>eval/precision</td><td>0.60436</td></tr><tr><td>eval/recall</td><td>0.35959</td></tr><tr><td>eval/runtime</td><td>63.4169</td></tr><tr><td>eval/samples_per_second</td><td>20.294</td></tr><tr><td>eval/steps_per_second</td><td>2.539</td></tr><tr><td>train/epoch</td><td>3.0</td></tr><tr><td>train/global_step</td><td>3303</td></tr><tr><td>train/learning_rate</td><td>1e-05</td></tr><tr><td>train/loss</td><td>0.0282</td></tr><tr><td>train/total_flos</td><td>143292425445210.0</td></tr><tr><td>train/train_loss</td><td>0.09467</td></tr><tr><td>train/train_runtime</td><td>3104.1516</td></tr><tr><td>train/train_samples_per_second</td><td>4.255</td></tr><tr><td>train/train_steps_per_second</td><td>1.064</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">pleasant-sweep-8</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/9k7iuegg</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_055827-9k7iuegg\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "best_trial = trainer.hyperparameter_search(\n",
    "    direction=\"maximize\",\n",
    "    backend=\"wandb\",\n",
    "    #hp_space=wandb_hp_space,\n",
    "    n_trials=8,\n",
    "    #keep_checkpoints_num=1,\n",
    "    scheduler=get_scheduler()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BestRun(run_id='kgy00di9', objective=2.423856367441647, hyperparameters={'learning_rate': 5.6309788887288427e-05, 'num_train_epochs': 4, 'per_device_train_batch_size': 8, 'seed': 25, 'assignments': {}, 'metric': 'eval/loss'}, run_summary=None)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Changes to your `wandb` environment variables will be ignored because your `wandb` session has already started. For more information on how to modify your settings with `wandb.init()` arguments, please refer to <a href='https://wandb.me/wandb-init' target=\"_blank\">the W&B docs</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Ignored wandb.init() arg project when running a sweep.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_182419-s5xh67q4</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4' target=\"_blank\">run_2_06_04_2023_11_44_pm</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.185, 'learning_rate': 4.35353358729308e-05, 'epoch': 0.91}\n",
      "{'eval_loss': 0.24270012974739075, 'eval_precision': 0.5070603337612324, 'eval_recall': 0.366079703429101, 'eval_f1': 0.4251883745963402, 'eval_accuracy': 0.9448933350433928, 'eval_runtime': 80.7282, 'eval_samples_per_second': 15.942, 'eval_steps_per_second': 1.994, 'epoch': 0.91}\n",
      "{'loss': 0.0794, 'learning_rate': 3.0760882858573164e-05, 'epoch': 1.81}\n",
      "{'eval_loss': 0.2586621344089508, 'eval_precision': 0.6069868995633187, 'eval_recall': 0.386468952734013, 'eval_f1': 0.47225368063420153, 'eval_accuracy': 0.9482707024069087, 'eval_runtime': 61.545, 'eval_samples_per_second': 20.912, 'eval_steps_per_second': 2.616, 'epoch': 1.81}\n",
      "{'loss': 0.0404, 'learning_rate': 1.798642984421554e-05, 'epoch': 2.72}\n",
      "{'eval_loss': 0.28830501437187195, 'eval_precision': 0.577023498694517, 'eval_recall': 0.40963855421686746, 'eval_f1': 0.47913279132791325, 'eval_accuracy': 0.9482279509212945, 'eval_runtime': 60.501, 'eval_samples_per_second': 21.272, 'eval_steps_per_second': 2.661, 'epoch': 2.72}\n",
      "{'loss': 0.02, 'learning_rate': 5.211976829857912e-06, 'epoch': 3.63}\n",
      "{'eval_loss': 0.321499228477478, 'eval_precision': 0.5828947368421052, 'eval_recall': 0.41056533827618164, 'eval_f1': 0.4817835780315388, 'eval_accuracy': 0.9486127142918216, 'eval_runtime': 69.095, 'eval_samples_per_second': 18.627, 'eval_steps_per_second': 2.33, 'epoch': 3.63}\n",
      "{'train_runtime': 3274.7055, 'train_samples_per_second': 5.378, 'train_steps_per_second': 0.673, 'train_loss': 0.07536158587668638, 'epoch': 4.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=2204, training_loss=0.07536158587668638, metrics={'train_runtime': 3274.7055, 'train_samples_per_second': 5.378, 'train_steps_per_second': 0.673, 'train_loss': 0.07536158587668638, 'epoch': 4.0})"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for n, v in best_trial.hyperparameters.items():\n",
    "    setattr(trainer.args, n, v)\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e273d782edb404484c46badb71ebe74",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/1.11k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\avanjavakam\\AppData\\Roaming\\Python\\Python39\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\avanjavakam\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
      "To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
      "  warnings.warn(message)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf3c8ca1bf7e445bb5a69b3e24d9913f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/266M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fcfd1fbbfb6148a194753612781363c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/320 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e02431c7f0c4c098809f28b42c1cd99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac6a04e0deee4483bfbacbd1cbde47a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "985c97e1ccd74225a93fe6d8a233c6a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# after pushing the best model to hugging face hub (kgy00di9)\n",
    "model = AutoModelForTokenClassification.from_pretrained(\"anudeepvanjavakam/distilbert_finetuned_wnut17_wandb_ner\")\n",
    "trainer = Trainer(model=model)\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"anudeepvanjavakam/distilbert_finetuned_wnut17_wandb_ner\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         word            tag\n",
      "0       [CLS]              O\n",
      "1       apple  B-corporation\n",
      "2          un              O\n",
      "3        ##ve              O\n",
      "4       ##ils              O\n",
      "5         all              O\n",
      "6           -              O\n",
      "7         new              O\n",
      "8         mac      B-product\n",
      "9      ##book      I-product\n",
      "10        air      I-product\n",
      "11          ,              O\n",
      "12      super              O\n",
      "13  ##charged              O\n",
      "14         by              O\n",
      "15        the              O\n",
      "16        new              O\n",
      "17         m2      B-product\n",
      "18       chip      I-product\n",
      "19      [SEP]              O\n"
     ]
    }
   ],
   "source": [
    "text=\"\"\"Apple unveils all-new MacBook Air, supercharged by the new M2 chip\"\"\"\n",
    "\n",
    "print(tag_sentence(text))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing other options for hypertuning (without population based training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Create sweep with ID: wx2o1mj0\n",
      "Sweep URL: https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0\n"
     ]
    }
   ],
   "source": [
    "# method\n",
    "sweep_config = {\n",
    "    'method': 'random'\n",
    "}\n",
    "\n",
    "\n",
    "# hyperparameters\n",
    "parameters_dict = {\n",
    "    'epochs': {\n",
    "        'value': 1\n",
    "        },\n",
    "    'batch_size': {\n",
    "        'values': [8, 64]\n",
    "        },\n",
    "    'learning_rate': {\n",
    "        'distribution': 'log_uniform_values',\n",
    "        'min': 1e-5,\n",
    "        'max': 5e-5\n",
    "    },\n",
    "    'weight_decay': {\n",
    "        'values': [0.0, 0.3]\n",
    "    },\n",
    "}\n",
    "\n",
    "\n",
    "sweep_config['parameters'] = parameters_dict\n",
    "sweep_id = wandb.sweep(sweep_config, project='reddit_product_tagging')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(config=None):\n",
    "  with wandb.init(config=config):\n",
    "    # set sweep configuration\n",
    "    config = wandb.config\n",
    "\n",
    "\n",
    "    # set training arguments\n",
    "    training_args = TrainingArguments(\n",
    "        output_dir='reddit_product_tagging-sweeps',\n",
    "\t      report_to='wandb',  # Turn on Weights & Biases logging\n",
    "        num_train_epochs=config.epochs,\n",
    "        learning_rate=config.learning_rate,\n",
    "        weight_decay=config.weight_decay,\n",
    "        per_device_train_batch_size=config.batch_size,\n",
    "        per_device_eval_batch_size=16,\n",
    "        save_strategy='epoch',\n",
    "        evaluation_strategy='epoch',\n",
    "        logging_strategy='epoch',\n",
    "        load_best_model_at_end=True,\n",
    "        disable_tqdm=False\n",
    "    )\n",
    "\n",
    "    # define training loop\n",
    "    trainer = Trainer(\n",
    "        # model,\n",
    "        model_init=model_init,\n",
    "        args=training_args,\n",
    "        train_dataset=tokenized_train_dataset,\n",
    "        eval_dataset=tokenized_wnut['test'],\n",
    "        tokenizer=tokenizer,\n",
    "        data_collator=data_collator,\n",
    "        compute_metrics=compute_metrics\n",
    "    )\n",
    "\n",
    "\n",
    "    # start training loop\n",
    "    trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: 70nqsf42 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 8\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 3.0218375436134713e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_133235-70nqsf42</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/70nqsf42' target=\"_blank\">pleasant-sweep-1</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/70nqsf42' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/70nqsf42</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "44567c0a96824856b28e08585d8340c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/551 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.2136, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71da773a2c0a42febd1904d9c0d4f545",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.24516281485557556, 'eval_precision': 0.5699404761904762, 'eval_recall': 0.35495829471733087, 'eval_f1': 0.4374643061107938, 'eval_accuracy': 0.9446795776153222, 'eval_runtime': 70.0451, 'eval_samples_per_second': 18.374, 'eval_steps_per_second': 1.156, 'epoch': 1.0}\n",
      "{'train_runtime': 772.4679, 'train_samples_per_second': 5.7, 'train_steps_per_second': 0.713, 'train_loss': 0.21362103913959704, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94468</td></tr><tr><td>eval/f1</td><td>0.43746</td></tr><tr><td>eval/loss</td><td>0.24516</td></tr><tr><td>eval/precision</td><td>0.56994</td></tr><tr><td>eval/recall</td><td>0.35496</td></tr><tr><td>eval/runtime</td><td>70.0451</td></tr><tr><td>eval/samples_per_second</td><td>18.374</td></tr><tr><td>eval/steps_per_second</td><td>1.156</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>551</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.2136</td></tr><tr><td>train/total_flos</td><td>53572044478050.0</td></tr><tr><td>train/train_loss</td><td>0.21362</td></tr><tr><td>train/train_runtime</td><td>772.4679</td></tr><tr><td>train/train_samples_per_second</td><td>5.7</td></tr><tr><td>train/train_steps_per_second</td><td>0.713</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">pleasant-sweep-1</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/70nqsf42' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/70nqsf42</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_133235-70nqsf42\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Sweep Agent: Waiting for job.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Job received.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: 7oazxgdk with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 8\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 1.0863756495194372e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_134650-7oazxgdk</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/7oazxgdk' target=\"_blank\">crimson-sweep-2</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/7oazxgdk' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/7oazxgdk</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ea7b607e8b046c4b98037edea7a6bdc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/551 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.2898, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4353b3d63334824a91c7260cd5b7f02",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.28670361638069153, 'eval_precision': 0.4385026737967914, 'eval_recall': 0.22798887859128822, 'eval_f1': 0.3, 'eval_accuracy': 0.9373263220896926, 'eval_runtime': 73.652, 'eval_samples_per_second': 17.474, 'eval_steps_per_second': 1.1, 'epoch': 1.0}\n",
      "{'train_runtime': 765.3589, 'train_samples_per_second': 5.753, 'train_steps_per_second': 0.72, 'train_loss': 0.28975097323935173, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.93733</td></tr><tr><td>eval/f1</td><td>0.3</td></tr><tr><td>eval/loss</td><td>0.2867</td></tr><tr><td>eval/precision</td><td>0.4385</td></tr><tr><td>eval/recall</td><td>0.22799</td></tr><tr><td>eval/runtime</td><td>73.652</td></tr><tr><td>eval/samples_per_second</td><td>17.474</td></tr><tr><td>eval/steps_per_second</td><td>1.1</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>551</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.2898</td></tr><tr><td>train/total_flos</td><td>53572044478050.0</td></tr><tr><td>train/train_loss</td><td>0.28975</td></tr><tr><td>train/train_runtime</td><td>765.3589</td></tr><tr><td>train/train_samples_per_second</td><td>5.753</td></tr><tr><td>train/train_steps_per_second</td><td>0.72</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">crimson-sweep-2</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/7oazxgdk' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/7oazxgdk</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_134650-7oazxgdk\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: psbmlki8 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 2.23088531403636e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_140042-psbmlki8</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/psbmlki8' target=\"_blank\">devout-sweep-3</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/psbmlki8' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/psbmlki8</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f077a4b8e4a4d3b9746f6c26cf4efc3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5624, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba54f09d803c456bb7a33962322feb1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.41149789094924927, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 69.0685, 'eval_samples_per_second': 18.634, 'eval_steps_per_second': 1.173, 'epoch': 1.0}\n",
      "{'train_runtime': 584.0377, 'train_samples_per_second': 7.539, 'train_steps_per_second': 0.118, 'train_loss': 0.5623834582342617, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.4115</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>69.0685</td></tr><tr><td>eval/samples_per_second</td><td>18.634</td></tr><tr><td>eval/steps_per_second</td><td>1.173</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.5624</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.56238</td></tr><tr><td>train/train_runtime</td><td>584.0377</td></tr><tr><td>train/train_samples_per_second</td><td>7.539</td></tr><tr><td>train/train_steps_per_second</td><td>0.118</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">devout-sweep-3</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/psbmlki8' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/psbmlki8</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_140042-psbmlki8\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: dtg9du4u with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 1.3689961336757494e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_141136-dtg9du4u</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/dtg9du4u' target=\"_blank\">peachy-sweep-4</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/dtg9du4u' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/dtg9du4u</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d51ed9c08714d9d9f14e0baeb5bfc08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7141, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ad265b7f28d4b10a699c374b40fb7a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.433150053024292, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 72.073, 'eval_samples_per_second': 17.857, 'eval_steps_per_second': 1.124, 'epoch': 1.0}\n",
      "{'train_runtime': 592.1385, 'train_samples_per_second': 7.436, 'train_steps_per_second': 0.117, 'train_loss': 0.7141017637391022, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.43315</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>72.073</td></tr><tr><td>eval/samples_per_second</td><td>17.857</td></tr><tr><td>eval/steps_per_second</td><td>1.124</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.7141</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.7141</td></tr><tr><td>train/train_runtime</td><td>592.1385</td></tr><tr><td>train/train_samples_per_second</td><td>7.436</td></tr><tr><td>train/train_steps_per_second</td><td>0.117</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">peachy-sweep-4</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/dtg9du4u' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/dtg9du4u</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_141136-dtg9du4u\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Sweep Agent: Waiting for job.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Job received.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: qe51vtj6 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 2.4870948756271965e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_142248-qe51vtj6</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/qe51vtj6' target=\"_blank\">blooming-sweep-5</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/qe51vtj6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/qe51vtj6</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca53caabc2964ce5a6d33a7128f3bb32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5349, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21b57f5552a04b10964877743557fe50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.39835095405578613, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 72.0787, 'eval_samples_per_second': 17.855, 'eval_steps_per_second': 1.124, 'epoch': 1.0}\n",
      "{'train_runtime': 593.4053, 'train_samples_per_second': 7.42, 'train_steps_per_second': 0.116, 'train_loss': 0.5348971270132756, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.39835</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>72.0787</td></tr><tr><td>eval/samples_per_second</td><td>17.855</td></tr><tr><td>eval/steps_per_second</td><td>1.124</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.5349</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.5349</td></tr><tr><td>train/train_runtime</td><td>593.4053</td></tr><tr><td>train/train_samples_per_second</td><td>7.42</td></tr><tr><td>train/train_steps_per_second</td><td>0.116</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">blooming-sweep-5</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/qe51vtj6' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/qe51vtj6</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_142248-qe51vtj6\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: gpo7cdfs with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 2.124585114733642e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_143355-gpo7cdfs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/gpo7cdfs' target=\"_blank\">proud-sweep-6</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/gpo7cdfs' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/gpo7cdfs</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7012e9ca4ea444aea218aa4d41a08a22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5747, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bba3fe5eb8c4fbe8d221133eff6f40f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.41408681869506836, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 73.625, 'eval_samples_per_second': 17.48, 'eval_steps_per_second': 1.1, 'epoch': 1.0}\n",
      "{'train_runtime': 591.5063, 'train_samples_per_second': 7.444, 'train_steps_per_second': 0.117, 'train_loss': 0.5747222900390625, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.41409</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>73.625</td></tr><tr><td>eval/samples_per_second</td><td>17.48</td></tr><tr><td>eval/steps_per_second</td><td>1.1</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.5747</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.57472</td></tr><tr><td>train/train_runtime</td><td>591.5063</td></tr><tr><td>train/train_samples_per_second</td><td>7.444</td></tr><tr><td>train/train_steps_per_second</td><td>0.117</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">proud-sweep-6</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/gpo7cdfs' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/gpo7cdfs</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_143355-gpo7cdfs\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: s472n8t5 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 2.6184993091948376e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_144457-s472n8t5</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s472n8t5' target=\"_blank\">frosty-sweep-7</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s472n8t5' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s472n8t5</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0207165970cc451f8aa7f52d459d47f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.5208, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15538292fcd74b9aaa642302bbe2e914",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3851180374622345, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 69.27, 'eval_samples_per_second': 18.579, 'eval_steps_per_second': 1.169, 'epoch': 1.0}\n",
      "{'train_runtime': 594.8042, 'train_samples_per_second': 7.402, 'train_steps_per_second': 0.116, 'train_loss': 0.5208257592242697, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.38512</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>69.27</td></tr><tr><td>eval/samples_per_second</td><td>18.579</td></tr><tr><td>eval/steps_per_second</td><td>1.169</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.5208</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.52083</td></tr><tr><td>train/train_runtime</td><td>594.8042</td></tr><tr><td>train/train_samples_per_second</td><td>7.402</td></tr><tr><td>train/train_steps_per_second</td><td>0.116</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">frosty-sweep-7</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s472n8t5' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s472n8t5</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_144457-s472n8t5\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: pdem1p2k with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 1.2702375305629806e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_145603-pdem1p2k</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/pdem1p2k' target=\"_blank\">royal-sweep-8</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/pdem1p2k' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/pdem1p2k</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9fb1f420296c40a59c48ead2c15be216",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7453, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bba3263078aa4e51b535712e9eb90cd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4373634159564972, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 70.502, 'eval_samples_per_second': 18.255, 'eval_steps_per_second': 1.149, 'epoch': 1.0}\n",
      "{'train_runtime': 582.5221, 'train_samples_per_second': 7.559, 'train_steps_per_second': 0.118, 'train_loss': 0.7452821040499038, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.43736</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>70.502</td></tr><tr><td>eval/samples_per_second</td><td>18.255</td></tr><tr><td>eval/steps_per_second</td><td>1.149</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.7453</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.74528</td></tr><tr><td>train/train_runtime</td><td>582.5221</td></tr><tr><td>train/train_samples_per_second</td><td>7.559</td></tr><tr><td>train/train_steps_per_second</td><td>0.118</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">royal-sweep-8</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/pdem1p2k' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/pdem1p2k</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_145603-pdem1p2k\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: ir6gwzwz with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 8\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 4.73865707184315e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_150659-ir6gwzwz</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/ir6gwzwz' target=\"_blank\">drawn-sweep-9</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/ir6gwzwz' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/ir6gwzwz</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "864f13d0eb494f0a97564ef1e6770714",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/551 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.2014, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d4ece16e85248d1816cfb0337a2aec3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.23708440363407135, 'eval_precision': 0.5646551724137931, 'eval_recall': 0.36422613531047265, 'eval_f1': 0.4428169014084507, 'eval_accuracy': 0.9450215895002352, 'eval_runtime': 71.383, 'eval_samples_per_second': 18.03, 'eval_steps_per_second': 1.135, 'epoch': 1.0}\n",
      "{'train_runtime': 771.6396, 'train_samples_per_second': 5.706, 'train_steps_per_second': 0.714, 'train_loss': 0.20136521299607957, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.94502</td></tr><tr><td>eval/f1</td><td>0.44282</td></tr><tr><td>eval/loss</td><td>0.23708</td></tr><tr><td>eval/precision</td><td>0.56466</td></tr><tr><td>eval/recall</td><td>0.36423</td></tr><tr><td>eval/runtime</td><td>71.383</td></tr><tr><td>eval/samples_per_second</td><td>18.03</td></tr><tr><td>eval/steps_per_second</td><td>1.135</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>551</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.2014</td></tr><tr><td>train/total_flos</td><td>53572044478050.0</td></tr><tr><td>train/train_loss</td><td>0.20137</td></tr><tr><td>train/train_runtime</td><td>771.6396</td></tr><tr><td>train/train_samples_per_second</td><td>5.706</td></tr><tr><td>train/train_steps_per_second</td><td>0.714</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">drawn-sweep-9</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/ir6gwzwz' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/ir6gwzwz</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_150659-ir6gwzwz\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: s5xh67q4 with config:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 64\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 1\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 1.1409156823373444e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \tweight_decay: 0.3\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.3"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>c:\\Users\\avanjavakam\\OneDrive - Moulton Niguel Water\\Documents\\R_Home\\Udacity_Data_Scientist_Nano_Degree\\lit_or_not_on_reddit\\wandb\\run-20230605_152056-s5xh67q4</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4' target=\"_blank\">stilted-sweep-10</a></strong> to <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>Sweep page: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View sweep at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/sweeps/wx2o1mj0</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'learning_rate' was locked by 'sweep' (ignored update).\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Config item 'weight_decay' was locked by 'sweep' (ignored update).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43c332bf864e4060ac7cf798d458314a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/69 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7945, 'learning_rate': 0.0, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "068fde3e48b6400f842629b6b89e8bbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/81 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "c:\\Users\\avanjavakam\\.conda\\envs\\lit_or_not_on_reddit\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.44381245970726013, 'eval_precision': 0.0, 'eval_recall': 0.0, 'eval_f1': 0.0, 'eval_accuracy': 0.9256124150314223, 'eval_runtime': 80.581, 'eval_samples_per_second': 15.972, 'eval_steps_per_second': 1.005, 'epoch': 1.0}\n",
      "{'train_runtime': 617.4248, 'train_samples_per_second': 7.131, 'train_steps_per_second': 0.112, 'train_loss': 0.7944799229718637, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>▁</td></tr><tr><td>eval/f1</td><td>▁</td></tr><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/precision</td><td>▁</td></tr><tr><td>eval/recall</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁</td></tr><tr><td>train/global_step</td><td>▁▁▁</td></tr><tr><td>train/learning_rate</td><td>▁</td></tr><tr><td>train/loss</td><td>▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/accuracy</td><td>0.92561</td></tr><tr><td>eval/f1</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.44381</td></tr><tr><td>eval/precision</td><td>0.0</td></tr><tr><td>eval/recall</td><td>0.0</td></tr><tr><td>eval/runtime</td><td>80.581</td></tr><tr><td>eval/samples_per_second</td><td>15.972</td></tr><tr><td>eval/steps_per_second</td><td>1.005</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>69</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>0.7945</td></tr><tr><td>train/total_flos</td><td>67396717715400.0</td></tr><tr><td>train/train_loss</td><td>0.79448</td></tr><tr><td>train/train_runtime</td><td>617.4248</td></tr><tr><td>train/train_samples_per_second</td><td>7.131</td></tr><tr><td>train/train_steps_per_second</td><td>0.112</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">stilted-sweep-10</strong> at: <a href='https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4' target=\"_blank\">https://wandb.ai/anudeepvanjavakam/reddit_product_tagging/runs/s5xh67q4</a><br/>Synced 6 W&B file(s), 0 media file(s), 7 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230605_152056-s5xh67q4\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.agent(sweep_id, train, count=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>eval/accuracy</th>\n",
       "      <th>eval/f1</th>\n",
       "      <th>eval/loss</th>\n",
       "      <th>eval/precision</th>\n",
       "      <th>eval/recall</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>epochs</th>\n",
       "      <th>eval_batch_size</th>\n",
       "      <th>eval_steps</th>\n",
       "      <th>evaluation_strategy</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>per_device_eval_batch_size</th>\n",
       "      <th>per_device_train_batch_size</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Sweep</th>\n",
       "      <th>train_batch_size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>floral-sweep-3 (chosen model)</td>\n",
       "      <td>95%</td>\n",
       "      <td>48%</td>\n",
       "      <td>32%</td>\n",
       "      <td>58%</td>\n",
       "      <td>41%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000056</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>3036</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>zesty-sweep-6</td>\n",
       "      <td>95%</td>\n",
       "      <td>48%</td>\n",
       "      <td>39%</td>\n",
       "      <td>56%</td>\n",
       "      <td>42%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>6402</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>colorful-sweep-2</td>\n",
       "      <td>95%</td>\n",
       "      <td>47%</td>\n",
       "      <td>33%</td>\n",
       "      <td>60%</td>\n",
       "      <td>39%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>3041</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>gentle-sweep-1</td>\n",
       "      <td>95%</td>\n",
       "      <td>48%</td>\n",
       "      <td>26%</td>\n",
       "      <td>54%</td>\n",
       "      <td>43%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000062</td>\n",
       "      <td>8</td>\n",
       "      <td>64</td>\n",
       "      <td>2155</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>kind-sweep-5</td>\n",
       "      <td>95%</td>\n",
       "      <td>47%</td>\n",
       "      <td>26%</td>\n",
       "      <td>58%</td>\n",
       "      <td>40%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000047</td>\n",
       "      <td>8</td>\n",
       "      <td>32</td>\n",
       "      <td>1667</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>pleasant-sweep-8</td>\n",
       "      <td>95%</td>\n",
       "      <td>45%</td>\n",
       "      <td>35%</td>\n",
       "      <td>60%</td>\n",
       "      <td>36%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000074</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>3166</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>celestial-sweep-7</td>\n",
       "      <td>95%</td>\n",
       "      <td>45%</td>\n",
       "      <td>25%</td>\n",
       "      <td>54%</td>\n",
       "      <td>38%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>8</td>\n",
       "      <td>64</td>\n",
       "      <td>1214</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>drawn-sweep-9</td>\n",
       "      <td>95%</td>\n",
       "      <td>44%</td>\n",
       "      <td>24%</td>\n",
       "      <td>56%</td>\n",
       "      <td>36%</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000047</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>835</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>pleasant-sweep-1</td>\n",
       "      <td>94%</td>\n",
       "      <td>44%</td>\n",
       "      <td>25%</td>\n",
       "      <td>57%</td>\n",
       "      <td>35%</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>855</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>crimson-sweep-2</td>\n",
       "      <td>94%</td>\n",
       "      <td>30%</td>\n",
       "      <td>29%</td>\n",
       "      <td>44%</td>\n",
       "      <td>23%</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>834</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>twilight-sweep-4</td>\n",
       "      <td>94%</td>\n",
       "      <td>31%</td>\n",
       "      <td>29%</td>\n",
       "      <td>41%</td>\n",
       "      <td>25%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>8</td>\n",
       "      <td>64</td>\n",
       "      <td>2670</td>\n",
       "      <td>ile8sxcl</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>amber-sweep-1</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>35%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>60.0</td>\n",
       "      <td>steps</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>1908</td>\n",
       "      <td>e2bagj2t</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>peachy-sweep-4</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>43%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>659</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>proud-sweep-6</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>41%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000021</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>660</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>royal-sweep-8</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>44%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>656</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>devout-sweep-3</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>41%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>650</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>blooming-sweep-5</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>40%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>670</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>frosty-sweep-7</td>\n",
       "      <td>93%</td>\n",
       "      <td>0%</td>\n",
       "      <td>39%</td>\n",
       "      <td>0%</td>\n",
       "      <td>0%</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>epoch</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>16</td>\n",
       "      <td>64</td>\n",
       "      <td>663</td>\n",
       "      <td>wx2o1mj0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Name eval/accuracy eval/f1 eval/loss   \n",
       "0   floral-sweep-3 (chosen model)           95%     48%       32%  \\\n",
       "1                   zesty-sweep-6           95%     48%       39%   \n",
       "2                colorful-sweep-2           95%     47%       33%   \n",
       "3                  gentle-sweep-1           95%     48%       26%   \n",
       "4                    kind-sweep-5           95%     47%       26%   \n",
       "5                pleasant-sweep-8           95%     45%       35%   \n",
       "6               celestial-sweep-7           95%     45%       25%   \n",
       "7                   drawn-sweep-9           95%     44%       24%   \n",
       "8                pleasant-sweep-1           94%     44%       25%   \n",
       "9                 crimson-sweep-2           94%     30%       29%   \n",
       "10               twilight-sweep-4           94%     31%       29%   \n",
       "11                  amber-sweep-1           93%      0%       35%   \n",
       "12                 peachy-sweep-4           93%      0%       43%   \n",
       "13                  proud-sweep-6           93%      0%       41%   \n",
       "14                  royal-sweep-8           93%      0%       44%   \n",
       "15                 devout-sweep-3           93%      0%       41%   \n",
       "16               blooming-sweep-5           93%      0%       40%   \n",
       "17                 frosty-sweep-7           93%      0%       39%   \n",
       "\n",
       "   eval/precision eval/recall  batch_size  epochs  eval_batch_size   \n",
       "0             58%         41%         NaN     NaN                8  \\\n",
       "1             56%         42%         NaN     NaN                8   \n",
       "2             60%         39%         NaN     NaN                8   \n",
       "3             54%         43%         NaN     NaN                8   \n",
       "4             58%         40%         NaN     NaN                8   \n",
       "5             60%         36%         NaN     NaN                8   \n",
       "6             54%         38%         NaN     NaN                8   \n",
       "7             56%         36%         8.0     1.0               16   \n",
       "8             57%         35%         8.0     1.0               16   \n",
       "9             44%         23%         8.0     1.0               16   \n",
       "10            41%         25%         NaN     NaN                8   \n",
       "11             0%          0%         NaN     NaN                8   \n",
       "12             0%          0%        64.0     1.0               16   \n",
       "13             0%          0%        64.0     1.0               16   \n",
       "14             0%          0%        64.0     1.0               16   \n",
       "15             0%          0%        64.0     1.0               16   \n",
       "16             0%          0%        64.0     1.0               16   \n",
       "17             0%          0%        64.0     1.0               16   \n",
       "\n",
       "    eval_steps evaluation_strategy  learning_rate  per_device_eval_batch_size   \n",
       "0        500.0               steps       0.000056                           8  \\\n",
       "1        500.0               steps       0.000037                           8   \n",
       "2        500.0               steps       0.000067                           8   \n",
       "3        500.0               steps       0.000062                           8   \n",
       "4        500.0               steps       0.000047                           8   \n",
       "5        500.0               steps       0.000074                           8   \n",
       "6        500.0               steps       0.000067                           8   \n",
       "7          NaN               epoch       0.000047                          16   \n",
       "8          NaN               epoch       0.000030                          16   \n",
       "9          NaN               epoch       0.000011                          16   \n",
       "10       500.0               steps       0.000008                           8   \n",
       "11        60.0               steps       0.000003                           8   \n",
       "12         NaN               epoch       0.000014                          16   \n",
       "13         NaN               epoch       0.000021                          16   \n",
       "14         NaN               epoch       0.000013                          16   \n",
       "15         NaN               epoch       0.000022                          16   \n",
       "16         NaN               epoch       0.000025                          16   \n",
       "17         NaN               epoch       0.000026                          16   \n",
       "\n",
       "    per_device_train_batch_size  Runtime     Sweep  train_batch_size  \n",
       "0                             8     3036  ile8sxcl                 8  \n",
       "1                             4     6402  ile8sxcl                 4  \n",
       "2                            16     3041  ile8sxcl                16  \n",
       "3                            64     2155  ile8sxcl                64  \n",
       "4                            32     1667  ile8sxcl                32  \n",
       "5                             4     3166  ile8sxcl                 4  \n",
       "6                            64     1214  ile8sxcl                64  \n",
       "7                             8      835  wx2o1mj0                 8  \n",
       "8                             8      855  wx2o1mj0                 8  \n",
       "9                             8      834  wx2o1mj0                 8  \n",
       "10                           64     2670  ile8sxcl                64  \n",
       "11                           16     1908  e2bagj2t                16  \n",
       "12                           64      659  wx2o1mj0                64  \n",
       "13                           64      660  wx2o1mj0                64  \n",
       "14                           64      656  wx2o1mj0                64  \n",
       "15                           64      650  wx2o1mj0                64  \n",
       "16                           64      670  wx2o1mj0                64  \n",
       "17                           64      663  wx2o1mj0                64  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Results and Comparison Table\n",
    "\n",
    "comparison_table = pd.read_csv(\"comparison_table_hyperparameter_tuning/wandb_export_2023-06-05T21_34_26.822-07_00.csv\")\n",
    "comparison_table"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "A low F1 score but high accuracy is observed. when it comes to the F1 score, which is a measure of the balance between precision and recall, DistilBERT may sometimes show slightly lower performance compared to the larger BERT model or other more complex models. This is because DistilBERT sacrifices some model capacity to achieve faster inference times and a smaller model size. Consequently, it may have slightly lower precision or recall compared to larger models, leading to a lower F1 score in some cases.\n",
    "\n",
    "Class imbalance: The dataset is imbalanced, meaning one class has significantly more samples than the other, the model might have a high accuracy by predicting the majority class correctly most of the time. However, it may struggle with the minority class, resulting in lower recall and F1 score for that class.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "- Recap of the this model objective: Find product mentions amongst posts and comments via named entity recognition (NER). This model can be used by the app to help users with queries such as \"Best 4K TV to buy\"\n",
    "\n",
    "- Methodology: a baseline model is developed for initial evaluation. A pre-trained DistilBERT model (token-classification) from Hugging Face Transformers which did not have product entities is fine tuned with the readily available wnut_17 data set to predict products. Hyperparameter tuning with \"Weights and Biases\" (provides tools to quickly track experiments, version and iterate on datasets, evaluate model performance, reproduce models, and visualize results) was performed to find the best model with an accuracy of 95% evaluated and tested on examples. This best model is pushed to the Hugging Face hub and can be called for inference through the app.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improvement\n",
    "\n",
    "Examine the data and check how many examples have each tags/token. Model needs more product data to be trained on.\n",
    "\n",
    "Cross-validation (with a stratified split since classes aren’t balanced) may help mitigate some of the problems that come from doing a train/test split with small datasets.\n",
    "\n",
    "Class imbalance: there is a class imbalance (model may predict the majority class “O” majority of the time) and needs to be addressed by employing techniques like oversampling the minority class, undersampling the majority class, or using class weighting.\n",
    "\n",
    "Consider other evaluation metrics.\n",
    "\n",
    "Fine-tuning with more reddit comments and product entities can help the model learn better representations and improve its performance on the task at hand."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgement:\n",
    "- https://huggingface.co/datasets/wnut_17\n",
    "- https://huggingface.co/learn/nlp-course/chapter7/2\n",
    "- https://towardsdatascience.com/named-entity-recognition-with-deep-learning-bert-the-essential-guide-274c6965e2d\n",
    "- https://towardsdatascience.com/build-a-named-entity-recognition-app-with-streamlit-f157672f867f\n",
    "- https://github.com/JINHXu/create-annotated-NER-dataset\n",
    "- https://wandb.ai/matt24/vit-snacks-sweeps/reports/Hyperparameter-Search-with-W-B-Sweeps-for-Hugging-Face-Transformer-Models--VmlldzoyMTUxNTg0\n",
    "- https://docs.wandb.ai/guides/integrations/huggingface#2-name-the-project\n",
    "- https://docs.wandb.ai/guides/sweeps/initialize-sweeps\n",
    "- https://wandb.ai/amogkam/transformers/reports/Hyperparameter-Optimization-for-Huggingface-Transformers--VmlldzoyMTc2ODI\n",
    "- https://huggingface.co/docs/transformers/hpo_train\n",
    "- https://docs.streamlit.io/library/advanced-features/caching\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lit_or_not_on_reddit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}