sample-notebooks/genomics-encode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ENCODE on Azure Genomics Data Lake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter notebook is a great tool for data scientists who is working on Genomics data analysis. We will demonstrate usage of Encyclopedia of DNA Elements (ENCODE) data from Azure Open Datasets.\n",
    "\n",
    "**Here is the coverage of this notebook:**\n",
    "\n",
    "1. Getting the ENCODE data from Azure Open Dataset\n",
    "2. Import the 'encode_file_manifest.tsv' to a table\n",
    "3. Checking the count of specific files\n",
    "\n",
    "**Dependencies:**\n",
    "\n",
    "This notebook requires the following libraries:\n",
    "\n",
    "- Azure storage `pip install azure-storage-blob==2.1.0`. Please visit [this page](https://github.com/Azure/azure-storage-python/wiki) for frequently encountered problem for this SDK.\n",
    "\n",
    "\n",
    "- Technical note: [Explore Azure Genomics Data Lake with Azure Storage Explorer](https://github.com/microsoft/genomicsnotebook/blob/main/docs/Genomics_Data_Lake_Azure_Storage_Explorer.pdf)\n",
    "\n",
    "**Important information: This notebook is using Python 3.6 kernel**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Getting the ENCODE data from Azure Open Dataset\n",
    "\n",
    "Several public genomics data has been uploaded as an Azure Open Dataset [here](https://azure.microsoft.com/services/open-datasets/catalog/). We create a blob service linked to this open datasets. You can find example of data calling procedure from Azure Open Dataset for `ENCODE` datasets in below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1.a.Install Azure Blob Storage SDK**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install azure-storage-blob==2.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1.b.Download the targeted file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import uuid\n",
    "import sys\n",
    "from azure.storage.blob import BlockBlobService, PublicAccess\n",
    "\n",
    "blob_service_client = BlockBlobService(account_name='datasetencode', sas_token='?sv=2019-10-10&si=prod&sr=c&sig=9qSQZo4ggrCNpybBExU8SypuUZV33igI11xw0P7rB3c%3D')     \n",
    "blob_service_client.get_blob_to_path('dataset', 'encode_file_manifest.tsv', './encode_file_manifest.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Import the 'encode_file_manifest.tsv' to a table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>status</th>\n",
       "      <th>file_format</th>\n",
       "      <th>file_type</th>\n",
       "      <th>assembly</th>\n",
       "      <th>award.rfa</th>\n",
       "      <th>output_type</th>\n",
       "      <th>output_category</th>\n",
       "      <th>file_size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>ENCODE</td>\n",
       "      <td>signal p-value</td>\n",
       "      <td>signal</td>\n",
       "      <td>6.206849e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>ENCODE</td>\n",
       "      <td>plus strand signal of all reads</td>\n",
       "      <td>signal</td>\n",
       "      <td>6.236199e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>ENCODE</td>\n",
       "      <td>signal p-value</td>\n",
       "      <td>signal</td>\n",
       "      <td>6.222111e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>ENCODE</td>\n",
       "      <td>signal p-value</td>\n",
       "      <td>signal</td>\n",
       "      <td>6.442427e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>ENCODE</td>\n",
       "      <td>signal p-value</td>\n",
       "      <td>signal</td>\n",
       "      <td>6.222841e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641541</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>hg19</td>\n",
       "      <td>ENCODE2</td>\n",
       "      <td>summed densities signal</td>\n",
       "      <td>signal</td>\n",
       "      <td>1.309956e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641542</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>hg19</td>\n",
       "      <td>ENCODE2</td>\n",
       "      <td>wavelet-smoothed signal</td>\n",
       "      <td>signal</td>\n",
       "      <td>1.015879e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641543</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>hg19</td>\n",
       "      <td>ENCODE2</td>\n",
       "      <td>wavelet-smoothed signal</td>\n",
       "      <td>signal</td>\n",
       "      <td>1.021096e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641544</th>\n",
       "      <td>released</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>bigWig</td>\n",
       "      <td>hg19</td>\n",
       "      <td>ENCODE2</td>\n",
       "      <td>signal</td>\n",
       "      <td>signal</td>\n",
       "      <td>1.781798e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641545</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>641546 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          status file_format file_type assembly award.rfa  \\\n",
       "0       released      bigWig    bigWig   GRCh38    ENCODE   \n",
       "1       released      bigWig    bigWig   GRCh38    ENCODE   \n",
       "2       released      bigWig    bigWig   GRCh38    ENCODE   \n",
       "3       released      bigWig    bigWig   GRCh38    ENCODE   \n",
       "4       released      bigWig    bigWig   GRCh38    ENCODE   \n",
       "...          ...         ...       ...      ...       ...   \n",
       "641541  released      bigWig    bigWig     hg19   ENCODE2   \n",
       "641542  released      bigWig    bigWig     hg19   ENCODE2   \n",
       "641543  released      bigWig    bigWig     hg19   ENCODE2   \n",
       "641544  released      bigWig    bigWig     hg19   ENCODE2   \n",
       "641545       NaN         NaN       NaN      NaN       NaN   \n",
       "\n",
       "                            output_type output_category     file_size  \n",
       "0                        signal p-value          signal  6.206849e+08  \n",
       "1       plus strand signal of all reads          signal  6.236199e+08  \n",
       "2                        signal p-value          signal  6.222111e+08  \n",
       "3                        signal p-value          signal  6.442427e+08  \n",
       "4                        signal p-value          signal  6.222841e+08  \n",
       "...                                 ...             ...           ...  \n",
       "641541          summed densities signal          signal  1.309956e+07  \n",
       "641542          wavelet-smoothed signal          signal  1.015879e+07  \n",
       "641543          wavelet-smoothed signal          signal  1.021096e+07  \n",
       "641544                           signal          signal  1.781798e+10  \n",
       "641545                              NaN             NaN           NaN  \n",
       "\n",
       "[641546 rows x 8 columns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# read encode_file_manifest.tsv into a dataframe\n",
    "\n",
    "metadata = pd.read_table('encode_file_manifest.tsv',sep='\\t')\n",
    "\n",
    "metadata.iloc[:,[1,2,3,4,5,10,11,12]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Checking the count of specific files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 641546 files in this dataset\n",
      "There are 41423 ENCODE award.rfa files in this dataset\n",
      "There are 22487 ENCODE2 award.rfa files in this dataset:\n",
      "There are 163495 hg.19 assembled files in this dataset\n",
      "There are 281223 GRCh38 assembled files in this dataset\n",
      "There are 74579 signal p-value output type in this dataset\n",
      "There are 142 wavelet-smoothed signal output type in this dataset\n",
      "There are 16 summed densities signal output type in this dataset\n"
     ]
    }
   ],
   "source": [
    "# let's take a quick look around\n",
    "\n",
    "num_entries = len(metadata)\n",
    "\n",
    "print(\"There are {} files in this dataset\".format(num_entries))\n",
    "\n",
    "num_ENCODE=metadata['award.rfa'].eq('ENCODE').sum()\n",
    "\n",
    "print(\"There are {} ENCODE award.rfa files in this dataset\".format(num_ENCODE))\n",
    "\n",
    "num_ENCODE2=metadata['award.rfa'].eq('ENCODE2').sum()\n",
    "\n",
    "print(\"There are {} ENCODE2 award.rfa files in this dataset:\".format(num_ENCODE2))\n",
    "\n",
    "num_hg19=metadata['assembly'].eq('hg19').sum()\n",
    "\n",
    "print(\"There are {} hg.19 assembled files in this dataset\".format(num_hg19))\n",
    "\n",
    "\n",
    "num_GRCh38=metadata['assembly'].eq('GRCh38').sum()\n",
    "\n",
    "print(\"There are {} GRCh38 assembled files in this dataset\".format(num_GRCh38))\n",
    "\n",
    "\n",
    "num_signal=metadata['output_type'].eq('signal p-value').sum()\n",
    "\n",
    "print(\"There are {} signal p-value output type in this dataset\".format(num_signal))\n",
    "\n",
    "\n",
    "num_wavelet=metadata['output_type'].eq('wavelet-smoothed signal').sum()\n",
    "\n",
    "print(\"There are {} wavelet-smoothed signal output type in this dataset\".format(num_wavelet))\n",
    "\n",
    "\n",
    "num_density=metadata['output_type'].eq('summed densities signal').sum()\n",
    "\n",
    "print(\"There are {} summed densities signal output type in this dataset\".format(num_density))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reference\n",
    "\n",
    "1. [ENCODE: Encyclopedia of DNA Elements](https://www.encodeproject.org)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}