4_Dashboard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "single-solid",
   "metadata": {},
   "source": [
    "# COVID-19 Twitter Final Project Dashboard\n",
    "> This interactive dashboard is one of the three major output components of our MACS 30122 final project. It is based on some of the analysis done by the other two parts, but also adds some extra visualization and story telling elements.\n",
    "\n",
    ">![Coronavirus banner](https://www.dfweyes.com//files/2020/04/coronavirus.jpg)\n",
    "\n",
    "-------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "blond-somalia",
   "metadata": {},
   "outputs": [],
   "source": [
    "####Notice####\n",
    "#1. please run 2_COVID_Tweets_word_frequency_analysis.ipynb before this notebook. \n",
    "#Since preprocessed tweets are stored with it and is called as stored value here to save running time.\n",
    "#2. please install voila before running this notebook. \n",
    "#If your Jupyter notebook is updated, you will be able to see a voila button in the tool bar.\n",
    "#Click this bar and you will be directed to the interactive dashboard.\n",
    "#Or you can type voila <path-to-notebook> 4_Dashboard.ipynb in the command line.\n",
    "#Or use Voilà within a pre-existing Jupyter server, first start the server, then go to the following URL: <url-of-my-server>/voila\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "agricultural-cycle",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import covid_data_analysis as cov\n",
    "import numpy as np\n",
    "\n",
    "# ploting packages\n",
    "import seaborn as sns\n",
    "import wordcloud\n",
    "import matplotlib.pyplot as plt\n",
    "import plotly.graph_objects as go\n",
    "from ipywidgets import interact\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "closed-jurisdiction",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df = pd.read_csv(\"data/merged_df_final.csv\")\n",
    "merged_df.fillna(\"NA\", inplace = True)\n",
    "covid_state_df = pd.read_csv(\"data/covid-data-by-state.csv\")\n",
    "covid_state_df = cov.clean_data(covid_state_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "straight-lesson",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_df_within_period(df, begin, end):\n",
    "    df = df[df.date >= begin]\n",
    "    df = df[df.date <= end]\n",
    "    return  df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stylish-klein",
   "metadata": {},
   "source": [
    "### State-level COVID numbers and Tweets:\n",
    "\n",
    "\n",
    "Users can identify an `US state` and a given `date` to see the State-level COVID number and tweets from the CDC on that date.\n",
    "\n",
    "Inputs:\n",
    "- quoted full name of an US state, with the first letter capitalized. `E.g., \"Alaska\"`\n",
    "- date of standard date format: `year/month/day`, ranging from 2020/03/10 to 2021/02/21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "classical-language",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3de5e9eabeaa4df7a0665dd1724b4633",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Text(value='Alaska', description='state'), Text(value='2021/02/19', description='date'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_cdc_tweets(merged_df, date):\n",
    "    tweet = merged_df[merged_df.date == date].CDC_tweet\n",
    "    num_t = 0\n",
    "    t_dedup = set()\n",
    "    for t in tweet:\n",
    "        if t != \"NA\" and t not in t_dedup:\n",
    "            num_t +=1\n",
    "            t_dedup.add(t)\n",
    "            display(HTML(\"<div style = 'background-color: #504e4e; padding: 30px '>\" +\n",
    "                         \"<span style='color: red; font-size:20px;'> CDC Tweet \" + str(num_t) + \":\" +\"</span>\" +\n",
    "                         \"<span style='color: #fff; font-size:15px;'>\" + str(t) +\"</span>\" +\n",
    "                         \"</div>\"))\n",
    "    if num_t == 0:\n",
    "        display(HTML(\"<div style = 'background-color: #504e4e; padding: 30px '>\" +\n",
    "                     \"<span style='color: #fff; font-size:15px;'> There are no tweets from CDC found on this day\" + \"</span>\" +\n",
    "                     \"</div>\"))\n",
    "\n",
    "def plot_gov_tweets(merged_df, date, state):\n",
    "    df_state = merged_df[merged_df.province_state == state]\n",
    "    tweet = df_state[df_state.date == date].governor_tweet\n",
    "    num_t = 0\n",
    "    t_dedup = set()\n",
    "    for t in tweet:\n",
    "        if t != \"NA\" and t not in t_dedup:\n",
    "            num_t +=1\n",
    "            t_dedup.add(t)\n",
    "            display(HTML(\"<div style = 'background-color: #504e4e; padding: 30px '>\" +\n",
    "                         \"<span style='color: red; font-size:20px;'> Governor Tweet \" + str(num_t) + \":\" +\"</span>\" +\n",
    "                         \"<span style='color: #fff; font-size:15px;'>\" + str(t) +\"</span>\" +\n",
    "                         \"</div>\"))\n",
    "    if num_t == 0:\n",
    "        display(HTML(\"<div style = 'background-color: #504e4e; padding: 30px '>\" +\n",
    "                     \"<span style='color: #fff; font-size:15px;'> There are no tweets from the state governor found on this day\" + \"</span>\" +\n",
    "                     \"</div>\"))\n",
    "\n",
    "def plot_cov_num_tweet(merged_df, state, date):\n",
    "    df_dt = merged_df[merged_df.date == date]\n",
    "    df = df_dt[df_dt.province_state == state]\n",
    "    confirm = df.confirmed_state.iloc[0]\n",
    "    deaths = df.deaths_state.iloc[0]\n",
    "    display(HTML(\"<div style = 'background-color: #fff; padding: 20px '>\" +\n",
    "                     \"<span style='color: black; font-size:22px;'> Confirmed number:\" + str(confirm) + \"</span>\" +\n",
    "                     \"<span style='color: red; font-size:22px;'> Deaths number:\" + str(deaths) + \"</span>\" +\n",
    "                     \"</div>\"))\n",
    "\n",
    "@interact(state = \"Alaska\", date = \"2021/02/19\")\n",
    "def cov_tweet_search(state, date):\n",
    "    plot_cov_num_tweet(merged_df, state, date)\n",
    "    plot_cdc_tweets(merged_df, date)\n",
    "    plot_gov_tweets(merged_df, date, state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aboriginal-algebra",
   "metadata": {},
   "source": [
    "-------------\n",
    "\n",
    "### COVID state map within customized period\n",
    "\n",
    "Users can input `start` and `end` date to identify a period of their interest, to explore the COVID trends of top `n` states by confirmed/deaths numbers. Update of the plots may take a while.\n",
    "\n",
    "Inputs:\n",
    "- begin and end date of standard date format: `year-month-day`, within the range from 2020-03-10 to 2021-02-21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "descending-asset",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4337b4188afb425b8d8c0d7cd65cd11f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Text(value='2020-03-10', description='begin'), Text(value='2021-02-21', description='end…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(begin = \"2020-03-10\", end = \"2021-02-21\")\n",
    "def plot_state_map(begin, end):\n",
    "    df = get_df_within_period(covid_state_df, begin, end)\n",
    "    cov.draw_state_heatmap(df, \"confirmed_state\")\n",
    "    cov.draw_state_heatmap(df, \"deaths_state\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "automotive-radius",
   "metadata": {},
   "source": [
    "-------------\n",
    "\n",
    "### Top hit states within customized period\n",
    "\n",
    "Users can input `start` and `end` date to identify a period of their interest, to explore the COVID trends of top `n` states by confirmed/deaths numbers. Update of the plots may take a while.\n",
    "\n",
    "Inputs:\n",
    "- date of standard date format: `year-month-day`, ranging from 2020-03-10 to 2021-02-21\n",
    "- integer n between `1 to 10`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "artistic-counter",
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_top_n_hardest_hit_state(df, var='confirmed_state', n=5):\n",
    "    '''\n",
    "    Draw the top n hardest hit state by death/confirmed \n",
    "    var(str): the variable to be drawn (deaths or confirmed)\n",
    "    '''\n",
    "    latest_date = sorted(df.date)[-1]\n",
    "    top_n_state = list(df[df['date'] == latest_date]\n",
    "         .sort_values(by=var,ascending=False)\n",
    "         .province_state[:n])\n",
    "\n",
    "    # create a data frame contain the time trend of the top n state\n",
    "    top_state_time_trend_df = df[df.province_state.isin(top_n_state)]\n",
    "\n",
    "    # clean the name\n",
    "    if var == 'confirmed_state':\n",
    "        col_name = 'Confirmed'\n",
    "    else:\n",
    "        col_name = 'Deaths'\n",
    "    \n",
    "    sns.set(rc={'figure.figsize':(20,15)})\n",
    "    \n",
    "    # draw the relplot\n",
    "    sns.relplot(x='date', y=var, kind='line',\n",
    "                data=top_state_time_trend_df,\n",
    "                hue='province_state', palette=\"rocket\")\n",
    "\n",
    "    plt.xticks(rotation=90)\n",
    "\n",
    "    if len(top_state_time_trend_df) > 10:\n",
    "        plt.xticks(ticks=np.arange(0, len(top_state_time_trend_df)/n,\n",
    "                step=len(top_state_time_trend_df)/n//10),\n",
    "                 rotation=90)\n",
    "\n",
    "    plt.xlabel('Date')\n",
    "    plt.ylabel('Number')  \n",
    "    plt.title(label=f\"Top {n} States by {col_name} Number\") \n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "powerful-replication",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "744f65e7df0c4482bed9c270dc561341",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Text(value='2020-03-10', description='begin'), Text(value='2021-02-21', description='end…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(begin = \"2020-03-10\", end = \"2021-02-21\", n = (1, 10))\n",
    "def plot_top_hitted_state(begin, end, n):\n",
    "    df = get_df_within_period(covid_state_df, begin, end)\n",
    "    draw_top_n_hardest_hit_state(df, \"confirmed_state\", n)\n",
    "    draw_top_n_hardest_hit_state(df, \"deaths_state\", n)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "greater-infrared",
   "metadata": {},
   "source": [
    "-------------\n",
    "\n",
    "### Word clouds based on CDC tweets of chosen season\n",
    "\n",
    "Users can input the `season` they are interested in and get the word clouds we made based on CDC tweets. Word clouds aid visualization of text data by effectively displaying `freqently mentioned words`. Users may be able identify some interesting patterns from our work. The generation of word clouds may take 1-2 seconds. \n",
    "\n",
    "Inputs:\n",
    "- season format: `year season`. You can choose from five seasons: \"21 spring\", \"20 spring\", \"20 summer\", \"20 fall\", \"20 winter\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "imported-assembly",
   "metadata": {},
   "outputs": [],
   "source": [
    "####please make share to run load_cdc_covid_tweets.ipynb before the flowing#####\n",
    "\n",
    "%store -r tweets01\n",
    "%store -r tweets02\n",
    "%store -r tweets03\n",
    "%store -r tweets04\n",
    "%store -r tweets0121\n",
    "\n",
    "season_dct = {\"20 spring\":tweets01, \"20 summer\": tweets02,\n",
    "              \"20 fall\":  tweets03, \"20 winter\":tweets04,\n",
    "              \"21 spring\": tweets0121}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "proud-blast",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8142c03d68c344748529701e29af786f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Text(value='21 spring', description='season'), Output()), _dom_classes=('widget-interact…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def grey_color_func(word, font_size, position,orientation,random_state=None, **kwargs):\n",
    "    return(\"hsl(360,100%%, %d%%)\" % np.random.randint(40,100))\n",
    "\n",
    "@interact(season = \"21 spring\")\n",
    "def plot_word_cloud(season):\n",
    "    tweets = season_dct[season]\n",
    "    text = ' '.join(tweets['normalized_tokens'].sum())\n",
    "    wc = wordcloud.WordCloud(width=1600, height=800,).generate(text)\n",
    "\n",
    "    wc.recolor(color_func = grey_color_func)\n",
    "    plt.figure(figsize=(18,12), facecolor='k')\n",
    "    plt.imshow(wc)\n",
    "    plt.axis(\"off\")\n",
    "    plt.tight_layout(pad=0)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hairy-reflection",
   "metadata": {},
   "source": [
    "--------------------\n",
    "\n",
    "   `Thanks for watching!`\n",
    "    \n",
    " We are:\n",
    "- Jinfei ZHU: Data Collection and COVID Data Analysis\n",
    "- Xi CHENG: CDC Tweets Word and Phrase Frequency Analysis\n",
    "- Boya FU: Record Linkage\n",
    "- Yile CHEN: Data Visualization and Interative Dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "blond-debut",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}