azure-storage-container-mount.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "5d26a3b4-6f85-40f9-9d7b-09cf7300adba",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Mounting Azure storage account containers in Databricks\n",
    "\n",
    "> Databricks recommends not to use mount points in a production setup for security reasons. These setup instructions here are provided as a quick solution for quickstart environments.\n",
    "\n",
    "## Steps to complete before running the notebook\n",
    "1. Search for \"Microsoft Entra ID\" in the Azure portal and select the corresponding service.\n",
    "2. Select \"Manage > App registrations\" and list \"All applications\".\n",
    "3. Add a new app registration, Account type \"Single tenant\", e.g. named \"DatabricksEntraIdApp\".\n",
    "4. Select \"Manage > Certificates & secrets\".\n",
    "5. Add new client secret to the app.\n",
    "6. Copy and temporarily store the secret value. It will be disabled only right after the creation of the client secret.\n",
    "7. Get \"Application (client) ID\" and \"Directory (tenant) Id\" of the app. See Overview.\n",
    "8. Create an Azure Key Vault.\n",
    "9. Add role \"Key Vault Administrator\" to your user.\n",
    "10. Add secrets, see keys in the notebook.\n",
    "11. Add Role \"Key Vault Administrator\" to app \"AzureDatabricks\".\n",
    "12. Add Secret Scope in Databricks via https://YOUR-DATABRICKS-HOST/#secrets/createScope, get DNS Name (Vault URI) and Resource ID from the Azure Key Vault properties.\n",
    "13. Add the created Microsoft Entra ID app as \"Storage Blob Data Contributor\" to the container(s).\n",
    "14. Change the name of the storage account and the list of the containers in the notebook.\n",
    "\n",
    "## Links to additional information\n",
    "- https://learn.microsoft.com/en-us/azure/databricks/getting-started/connect-to-azure-storage\n",
    "- https://learn.microsoft.com/en-us/answers/questions/1290398/not-able-to-create-secrete-in-azure-key-valut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "c84c020d-ebd6-434d-a14b-64d2aa9e7e8c",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Set the name of the storage account and a list of containers to be mounted.\n",
    "adls_account_name = \"STORAGE_ACCOUNT\"\n",
    "adls_containers = [\"CONTAINER-1\", \"CONTAINER-2\"]\n",
    "\n",
    "application_id = dbutils.secrets.get(scope=\"kv-dbw-secrets\", key=\"sp-client-id\")          # Application (client) ID\n",
    "authentication_key = dbutils.secrets.get(scope=\"kv-dbw-secrets\", key=\"sp-client-secret\")  # Client secret value\n",
    "tenant_id = dbutils.secrets.get(scope=\"kv-dbw-secrets\", key=\"sp-tenant-id\")               # Directory (tenant) ID\n",
    "\n",
    "configs = {\"fs.azure.account.auth.type\": \"OAuth\",\n",
    "           \"fs.azure.account.oauth.provider.type\": \"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider\",\n",
    "           \"fs.azure.account.oauth2.client.id\": application_id,\n",
    "           \"fs.azure.account.oauth2.client.secret\": authentication_key,\n",
    "           \"fs.azure.account.oauth2.client.endpoint\": f\"https://login.microsoftonline.com/{tenant_id}/oauth2/token\"}\n",
    "\n",
    "for adls_container_name in adls_containers:\n",
    "  mount_point = f\"/mnt/{adls_account_name}/{adls_container_name}\"\n",
    "  source = f\"abfss://{adls_container_name}@{adls_account_name}.dfs.core.windows.net/\"\n",
    "  \n",
    "  # Mount only if the directory is not already mounted\n",
    "  if not any(mount.mountPoint == mount_point for mount in dbutils.fs.mounts()):\n",
    "    dbutils.fs.mount(\n",
    "      source = source,\n",
    "      mount_point = mount_point,\n",
    "      extra_configs = configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "bf44f312-f270-4c14-90e2-f6fc194f38b7",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "dbutils.notebook.exit('Azure datalake storage containers mounted.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f5531912-76e4-404c-9e70-9cf71ca21c16",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "for adls_container_name in adls_containers:\n",
    "  dbutils.fs.ls(f\"/mnt/{adls_account_name}/{adls_container_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "de94c183-3428-40bd-8d19-5a6dfdeeb0f6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "dbutils.fs.mounts()"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "azure-storage-container-mount",
   "widgets": {}
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}