sample-notebooks/1000-genomes_Azure_Genomics_Data_Lake.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1000 Genomes Project's data on Azure Genomics Data Lake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter notebook is a great tool for data scientists who is working on Genomics data analysis. We will demonstrate Azure Jupyter notebook usage via GATK and Picard with Azure Open Dataset. \n",
    "\n",
    "**Here is the coverage of this notebook:**\n",
    "\n",
    "1. Download the specific data from Azure Genomics Data Lake\n",
    "2. BuildBamIndex (Picard)\n",
    "\n",
    "**Dependencies:**\n",
    "\n",
    "This notebook requires the following libraries:\n",
    "\n",
    "- Azure storage `pip install azure-storage-blob==2.1.0`. Please visit [this page](https://github.com/Azure/azure-storage-python/wiki) for frequently encountered problem for this SDK.\n",
    "\n",
    "\n",
    "- Genome Analysis Toolkit (GATK) (*Users need to download GATK from Broad Institute's webpage into the same compute environment with this notebook: https://github.com/broadinstitute/gatk/releases*)\n",
    "\n",
    "- Technical note: [Explore Azure Genomics Data Lake with Azure Storage Explorer](https://github.com/microsoft/genomicsnotebook/blob/main/docs/Genomics_Data_Lake_Azure_Storage_Explorer.pdf)\n",
    "\n",
    "**Important information: This notebook is using Python 3.6 kernel**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Getting the 1000 Genomes Project's data from Azure Open Dataset\n",
    "\n",
    "Several public genomics data has been uploaded as an Azure Open Dataset [here](https://azure.microsoft.com/services/open-datasets/catalog/). We create a blob service linked to this open datasets. You can find example of data calling procedure from Azure Open Dataset for `1000 Genomes Project` datasets in below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1.a.Install Azure Blob Storage SDK**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install azure-storage-blob==2.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1.b.Download the targeted file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import uuid\n",
    "import sys\n",
    "from azure.storage.blob import BlockBlobService, PublicAccess\n",
    "\n",
    "blob_service_client = BlockBlobService(account_name='dataset1000genomes', sas_token='sv=2019-10-10&si=prod&sr=c&sig=9nzcxaQn0NprMPlSh4RhFQHcXedLQIcFgbERiooHEqM%3D')     \n",
    "blob_service_client.get_blob_to_path('dataset/phase3/data/HG00096/alignment', 'HG00096.chrom11.ILLUMINA.bwa.GBR.low_coverage.20120522.bam', './HG00096.chrom11.ILLUMINA.bwa.GBR.low_coverage.20120522.bam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. BuildBamIndex (Picard)\n",
    "Generates a BAM index \".bai\" file. This tool creates an index file for the input BAM that allows fast look-up of data in a BAM file, lke an index on a database. Note that this tool cannot be run on SAM files, and that the input BAM file must be sorted in coordinate order[1].\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./gatk BuildBamIndex I=HG00096.chrom11.ILLUMINA.bwa.GBR.low_coverage.20120522.bam O=HG00096.chrom11.ILLUMINA.bwa.GBR.low_coverage.20120522.bam.bai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "1. BuildBamIndex: http://broadinstitute.github.io/picard/command-line-overview.html#BuildBamIndex\n",
    "2. 1000 Genomes Project: https://www.internationalgenome.org/\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}