Evaluation_Example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "36735b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import (\n",
    "    AutoTokenizer,\n",
    "    LEDForConditionalGeneration,\n",
    ")\n",
    "from datasets import load_dataset, load_metric\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9f59338",
   "metadata": {},
   "source": [
    "First, we load the **Multi-news** dataset from huggingface dataset hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5faeeffc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset multi_news/default (download: 245.06 MiB, generated: 667.74 MiB, post-processed: Unknown size, total: 912.80 MiB) to /home/wenx/.cache/huggingface/datasets/multi_news/default/1.0.0/2e145a8e21361ba4ee46fef70640ab946a3e8d425002f104d2cda99a9efca376...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d83cdf56f2a4ee0bea672c1f8f5264d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/257M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset multi_news downloaded and prepared to /home/wenx/.cache/huggingface/datasets/multi_news/default/1.0.0/2e145a8e21361ba4ee46fef70640ab946a3e8d425002f104d2cda99a9efca376. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1daf64cd6b7c4556b9d695ae9d6add33",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset=load_dataset('multi_news')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9137321",
   "metadata": {},
   "source": [
    "Then we load the fine-tuned PRIMERA model, please download [it](https://storage.googleapis.com/primer_summ/PRIMER_multinews.tar.gz) to your local computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "890f434e",
   "metadata": {},
   "outputs": [],
   "source": [
    "PRIMER_path='../github/PRIMER/PRIMER_multinews_hf'\n",
    "TOKENIZER = AutoTokenizer.from_pretrained(PRIMER_path)\n",
    "MODEL = LEDForConditionalGeneration.from_pretrained(PRIMER_path)\n",
    "MODEL.gradient_checkpointing_enable()\n",
    "PAD_TOKEN_ID = TOKENIZER.pad_token_id\n",
    "DOCSEP_TOKEN_ID = TOKENIZER.convert_tokens_to_ids(\"<doc-sep>\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd5aabfc",
   "metadata": {},
   "source": [
    "We then define the functions to pre-process the data, as well as the function to generate summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7bfecd47",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_document(documents):\n",
    "    input_ids_all=[]\n",
    "    for data in documents:\n",
    "        all_docs = data.split(\"|||||\")[:-1]\n",
    "        for i, doc in enumerate(all_docs):\n",
    "            doc = doc.replace(\"\\n\", \" \")\n",
    "            doc = \" \".join(doc.split())\n",
    "            all_docs[i] = doc\n",
    "\n",
    "        #### concat with global attention on doc-sep\n",
    "        input_ids = []\n",
    "        for doc in all_docs:\n",
    "            input_ids.extend(\n",
    "                TOKENIZER.encode(\n",
    "                    doc,\n",
    "                    truncation=True,\n",
    "                    max_length=4096 // len(all_docs),\n",
    "                )[1:-1]\n",
    "            )\n",
    "            input_ids.append(DOCSEP_TOKEN_ID)\n",
    "        input_ids = (\n",
    "            [TOKENIZER.bos_token_id]\n",
    "            + input_ids\n",
    "            + [TOKENIZER.eos_token_id]\n",
    "        )\n",
    "        input_ids_all.append(torch.tensor(input_ids))\n",
    "    input_ids = torch.nn.utils.rnn.pad_sequence(\n",
    "        input_ids_all, batch_first=True, padding_value=PAD_TOKEN_ID\n",
    "    )\n",
    "    return input_ids\n",
    "\n",
    "\n",
    "def batch_process(batch):\n",
    "    input_ids=process_document(batch['document'])\n",
    "    # get the input ids and attention masks together\n",
    "    global_attention_mask = torch.zeros_like(input_ids).to(input_ids.device)\n",
    "    # put global attention on <s> token\n",
    "\n",
    "    global_attention_mask[:, 0] = 1\n",
    "    global_attention_mask[input_ids == DOCSEP_TOKEN_ID] = 1\n",
    "    generated_ids = MODEL.generate(\n",
    "        input_ids=input_ids,\n",
    "        global_attention_mask=global_attention_mask,\n",
    "        use_cache=True,\n",
    "        max_length=1024,\n",
    "        num_beams=5,\n",
    "    )\n",
    "    generated_str = TOKENIZER.batch_decode(\n",
    "            generated_ids.tolist(), skip_special_tokens=True\n",
    "        )\n",
    "    result={}\n",
    "    result['generated_summaries'] = generated_str\n",
    "    result['gt_summaries']=batch['summary']\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "250f3053",
   "metadata": {},
   "source": [
    "Next, we simply run the model on 10 data examples (or any number of examples you want)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "631ead96",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "282e6e400d8745859d31eb1e7b9d28aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "data_idx = random.choices(range(len(dataset['test'])),k=10)\n",
    "dataset_small = dataset['test'].select(data_idx)\n",
    "result_small = dataset_small.map(batch_process, batched=True, batch_size=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab2dd96d",
   "metadata": {},
   "source": [
    "After getting all the results, we load the evaluation metric. \n",
    "\n",
    "\n",
    "(Note in the original code, we didn't use the default aggregators, instead, we simply take average over all the scores.\n",
    "We simply use 'mid' in this notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d8f31905",
   "metadata": {},
   "outputs": [],
   "source": [
    "rouge = load_metric(\"rouge\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "81814601",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['– Amazon wants to be your go-to for last-minute deals this holiday season, so it\\'s offering a little something extra for brick-and-mortar stores, TechCrunch reports. The online retailer is offering users of its Price Check app up to $5 off any product they buy at a brick-and-mortar store if they use the app to scan the barcode on the item, take a picture, or type in the product\\'s name, the Verge reports. Amazon usually has lower prices than brick-and-mortar stores, \"so this is just the cherry on top,\" one analyst says. \"The ability to check prices on your mobile phone when you’re in a physical retail store is changing the way people shop,\" an Amazon mobile director says. \"Price transparency means that you can save money on the products you want and that’s a great thing for customers. Price Check in-store deals are another incentive to shop smart this holiday season.\"',\n",
       " '– A day care in San Antonio has been shut down after an infant was allegedly bitten 27 times by another child, KENS 5 reports. According to the San Antonio Express-News, the baby had bite marks on his back, eyes, cheeks, legs, stomach, and more. The parents of the baby say the day care operator waited four hours to call them and only did so when the injuries didn\\'t go away. The day care operator says she didn\\'t know she needed a license to operate. She says she called the baby\\'s parents within minutes of finding the bite marks. A neighbor tells KENS 5 he took his daughter to the day care for a year and never had any issues. A second parent says she had taken her child to the day care for several years and the incident is \"being blown out of proportion.\" Child Care Licensing ordered the homeowner to cease operations Monday after an investigation revealed she had been operating out of a home on Bentwood View without a license. Investigators found eight children in the home and determined it had been in operation for eight years. Investigators shut down the day care operation Monday. A Child Protective Services spokeswoman tells KENS 5 the operator has expressed an interest in getting a license and reopening the operation. She would be required to complete several steps, including background checks and CPR training for all employees.',\n",
       " '– Internet service providers like Comcast, AT&T, and Time Warner can now do as they please, according to a federal appeals court decision that could have a huge impact on the Internet as a whole. The court struck down the FCC\\'s \"open Internet\" rules, which had been put in place in 2010 to ensure that Internet providers wouldn\\'t be able to slow or block traffic from sites they didn\\'t like or charge for access to certain parts of the internet. The ruling could mean that Netflix, YouTube, and other content-heavy websites could be forced to pay a fee to Internet service providers to ensure quality service, the Wall Street Journal reports. Here\\'s what else you need to know about the decision: The ruling is a \"potentially crippling defeat for net neutrality advocates,\" writes Chris Morran at BuzzFeed. ISPs \"can already make gobs of money by charging the rich a ton for high-quality Internet while leaving the rest of America with subpar (or no) service,\" he writes. \"Now, with net neutrality gone, ISPs will be able to make even more money off their existing customer base. They won\\'t need to improve service or bring broadband to rural areas because they\\'ll be able to keep growing (financially, at least) by charging content providers more for faster delivery and charging customers more for faster access.\" \"We may be entering the era of sponsored data—the era of an internet that we don\\'t directly pay for, but that we also don\\'t control. It\\'s the old net neutrality nightmare, in other words, disguised as a gift,\" writes Chris Matyszczyk at the Los Angeles Times. \"The internet would become a club with various VIP sections, arbitrarily laid out to benefit internet providers,\" he writes. \"It\\'s an intoxicating pitch, and one you\\'ll hear soon.\" \"While the court\\'s ruling will worry and anger advocates of an open Internet, it\\'s federal regulators, not the judges, who are to blame,\" writes Troy Wolverton at the San Jose Mercury News. \"Their subservience to the big telecommunications companies and timidity in writing the rules governing Internet traffic led directly to the court\\'s decision.\"',\n",
       " '– Two young men in Germany died from heart complications linked to marijuana use, according to a new study. Researchers found traces of the main active ingredient in marijuana in the blood and brain of both men, LiveScience reports. In one case, a 23-year-old man without a history of health problems collapsed on public transportation and died after 40 minutes of unsuccessful resuscitation. In the other, a 28-year-old man was found dead at home with marijuana rolling paper, ashtray, and a bag containing remnants of the drug next to his body. Both deaths were related to cardiovascular complications. \"We assume the deaths of these two young men occurred due to arrhythmias evoked by smoking cannabis,\" the researchers write in the Journal of the American Heart Association. \"This unexpected finding deserved to be further analyzed, especially given that the medicinal use of marijuana has become more prevalent and some governments are legalizing its use.\" The researchers declined an interview request from LiveScience, citing media response and \"some quite unpleasant reactions from individuals.\" A toxicologist at the San Diego County Medical Examiner\\'s Office says it\\'s \"important that people realize that any drug can have harmful effects.\" But the head of the German Association for Drugs and Addiction says marijuana \"does not paralyze the breathing or the heart,\" and \"deaths due to cannabis use are usually accidents that are not caused by the substance, but to the circumstances of use.\"',\n",
       " '– A Tesla \"whistleblower\" is suing his former employer, claiming it defamed him when it told the media he threatened a mass shooting, the Guardian reports. According to Business Insider, Martin Tripp, a former process technician at Tesla\\'s Gigafactory in Nevada, \"observed that the waste and scrap levels and non-conforming materials generated by Counterdefendant dramatically increased\" after a March 2018 goal of ramping up to 2,500 Model 3s and then 5,000 by July 2018. Tripp says he \"raised his concerns about the seemingly exorbitant amounts of scrap with his managers and supervisors but no action was taken.\" On June 20, Tesla sued Tripp, accusing him of hacking into the company\\'s manufacturing operating system and transferring secret information to third parties. Tesla also claimed a friend of Tripp threatened to \"shoot up\" the Gigafactory, but law enforcement looked into it and found no credible threat. On Tuesday, Tripp filed a counterclaim against Tesla and Elon Musk, arguing that Tesla defamed him by publishing false and defamatory statements about him. Tripp says he didn\\'t even know how to code. He\\'s seeking at least $1 million and punitive damages for defamation, invasion of privacy, and intentional infliction of emotional distress. “There were very suspicious, very contradictory stories that were advanced by Tesla,\" Tripp\\'s attorney says. \"We’re going to be interested in getting to the bottom of how the story was concocted.”',\n",
       " '– With the government shutdown entering its third day, President Trump took to Twitter on Sunday to blast Democrats for their \"far left base\" demands on immigration. \"The Democrats are turning down services and security for citizens in favor of services and security for non-citizens. Not good!\" he wrote, per the Washington Post. He also suggested that Republicans use the \"Nuclear Option\" to change Senate rules and pass a spending bill with a simple majority instead of the 60 votes needed to pass most legislation. \"Great to see how hard Republicans are fighting for our Military and Safety at the Border,\" he wrote. \"The Dems just want illegal immigrants to pour into our nation unchecked. If stalemate continues, Republicans should go to 51% (Nuclear Option) and vote on real, long term budget, no C.R.\\'s!\"',\n",
       " '– An 18-year-old college student\\'s leg was amputated after he stepped on a homemade firecracker in New York City\\'s Central Park on Sunday, the New York Daily News reports. According to CBS New York, the victim, identified as Connor Golden of Virginia, was climbing a rock structure with two friends when he stepped on something that exploded. “I was walking in front of him and suddenly heard this extremely loud explosion directly behind me,\" Thomas Hinds tells the New York Post. \"When I turned around, I saw Connor lying there, his foot completely gone. It was insane. He was moaning and saying, ‘Get help.’” The bomb squad was called in to investigate, and they determined the blast was caused by \"explosive experimentation,\" or the making of one\\'s own fireworks. “It is not unusual for the public to make or try to create homemade fireworks around the Fourth of July,” Lt. Mark Torre of the NYPD Bomb Squad tells the Post. “This is a time of a year where typically we will see a lot of experimentation—explosive experimenters, if you will. Their goal is to make a loud noise, maybe make a flash.” Torre says there is no evidence the homemade firecracker Golden stepped on was meant to cause harm. “There is some forensic evidence that indicates that it was not meant go off by someone stepping on it,” Torre says. “It was there longer than a day.” Police say they don\\'t believe Golden or his friends were involved in making the explosive.',\n",
       " '– Mark Zuckerberg and other Silicon Valley bigwigs are distancing themselves from Marc Andreessen after the Facebook board member appeared to support \"Internet colonialism\" in a series of now-deleted tweets Tuesday night, Business Insider reports. \"Anti-colonialism has been economically catastrophic for the Indian people for decades. Why stop now?\" the Wall Street Journal quotes the now-deleted tweet as saying. \"I want to respond to Marc Andreessen\\'s comments about India yesterday,\" Zuckerberg wrote on Facebook. \"I found the comments deeply upsetting, and they do not represent the way Facebook or I think at all.\" \"We strongly reject the sentiments expressed by Marc Andreessen last night regarding India,\" Facebook said in a statement. Andreessen apologized \"in full and without reservation.\"',\n",
       " '– The Miami Dolphins have fired offensive line coach Jim Turner and head trainer Kevin O\\'Neill over allegations of misconduct in the locker room, reports ESPN. \"The language and behavior as described in the Ted Wells report are against the core values of our organization,\" said team owner Stephen Ross in a statement. \"Jim Turner and Kevin O\\'Neill are good people who care a great deal about their profession and the players whom they serve, but both exhibited poor judgment at times which led me to this conclusion.\" O\\'Neill had been in charge of the team\\'s training staff. The Dolphins will hold a press conference today to discuss the results of the investigation.',\n",
       " '– With the clock ticking on the Sept. 30 deadline to avoid a government shutdown, House Republicans made good on their threat to defund Planned Parenthood today, reports the Washington Post. The vote was 241-187, with almost all Republicans in favor and two Democrats in favor, reports the AP. The legislation would put a one-year moratorium on federal funding for the group, which Republicans say will allow for a thorough investigation of its practices. Any funding that remains would be sent to clinics that don\\'t offer abortions. \"Most people think that is common sense,\" said Rep. Diane Black, the sponsor of the defunding bill. \"If there is reason to investigate, then there is reason to withhold taxpayer dollars during that period of time. The American taxpayer has been clear for a very long time that they do not want federal funds spent on abortion.\" Democrats pointed out that Planned Parenthood has long been prohibited from spending federal money on abortions thanks to appropriations riders dating back to the 1970s. \"This bill is dumb, it’s foolish, and it’s mean-spirited,\" said Rep. Lois Frankel. \"If you want to have a truthful debate, then let’s talk about the 400,000 Pap smears, the 500,000 breast exams, the 4.5 million STD and HIV tests that Planned Parenthood does each year.\" The House also passed the Born-Alive Abortion Survivors Protection Act, which would impose criminal penalties on medical personnel who fail to aid an infant born after an attempted abortion.']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_small['generated_summaries']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8d9923d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score(precision=0.509437078378281, recall=0.43832461548851936, fmeasure=0.4644188580686355)\n",
      "Score(precision=0.17689604682544763, recall=0.14564519595131636, fmeasure=0.1581222605371442)\n",
      "Score(precision=0.2362355904256852, recall=0.19669444890277293, fmeasure=0.21194685290367665)\n"
     ]
    }
   ],
   "source": [
    "score=rouge.compute(predictions=result_small[\"generated_summaries\"], references=result_small[\"gt_summaries\"])\n",
    "print(score['rouge1'].mid)\n",
    "print(score['rouge2'].mid)\n",
    "print(score['rougeL'].mid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6e5cdd3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c2c76256",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4496, 1390, 2088, 2130, 1604]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.choices(range(5000),k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e763cc4",
   "metadata": {},
   "source": [
    "– Facebook removed a photo of two men kissing in protest of a London pub’s decision to eject a same-sex couple for kissing, reports the America Blog. “Shares that contain nudity, or any kind of graphic or sexually suggestive content, are not permitted on Facebook,” the administrators of the Dangerous Minds Facebook page said in an email. The decision to remove the photo has prompted scores of people to post their own pictures of same-sex couples kissing in protest— dozens in the last few hours alone."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc3708f8",
   "metadata": {},
   "source": [
    "– Facebook has removed a photo from a protest page for a gay pub that booted a same-sex couple for kissing, USA Today reports. The Dangerous Minds Facebook page was trying to promote a “gay kiss-in” demonstration in London to protest the pub. The page used a photo of two men kissing to promote the event. But Facebook quickly removed the photo, saying in an email, “Shares that contain nudity, or any kind of graphic or sexually suggestive content, are not permitted on Facebook.” The decision to remove the photo has prompted scores of people to post their own pictures of same-sex couples kissing in protest— dozens in the last few hours alone."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}