tutorials-and-examples/how-tos/TroubleShootingNotebooks.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Troubleshooting Microsoft Sentinel Notebooks\n",
        "\n",
        "If you are having trouble with Jupyter notebooks run this notebook to help\n",
        "identify where the problem might be.\n",
        "\n",
        "Select the notebook menu item `Cell->Run All` - check for any warnings or errors.\n",
        "\n",
        "Read the text above the cell(s) that produce errors - the text\n",
        "contains links to resources that describe how to fix the error.\n",
        "\n",
        "> **Important**: you can run the cells individually but please run the\n",
        "> Python Version Check cell first, since this contains some function\n",
        "> definitions used by the other cells."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Python Version Check\n",
        "\n",
        "> **Note**\n",
        "> You can set the default Python version in Azure Notebooks project settings.\n",
        "> \n",
        "> For details on how to do this see [AzureNotebooks-ConfigurePythonVersion](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/AzureNotebooks-ConfigurePythonVersion.ipynb)\n",
        ">\n",
        "> If you are using a Data Science Virtual Machine as your\n",
        "> Azure Notebooks compute you should read [Provisioning a DSVM](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/Provisioning%20DSVM.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "gather": {
          "logged": 1618341999310
        }
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "#### Checking Python version..."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<h4><font color='blue'>Python version 3.7.10 - OK</font></h4>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import sys\n",
        "from IPython.display import display, HTML, Markdown\n",
        "MIN_REQ_PYTHON = (3, 10)\n",
        "\n",
        "errors = []\n",
        "warns = []\n",
        "info = []\n",
        "def setup_err(mssg):\n",
        "    display(Markdown(\"<h3><font color='red'>Setup Error</font></h3>\"))\n",
        "    display(Markdown(\"<h4><font color='red'>%s</font></h4>\" % mssg))\n",
        "    errors.append(mssg)\n",
        "\n",
        "def setup_ok(mssg):\n",
        "    display(Markdown(\"<h4><font color='blue'>%s - OK</font></h4>\" % mssg))\n",
        "    info.append(mssg)\n",
        "\n",
        "def setup_warn(mssg):\n",
        "    display(Markdown(\"<h4><font color='orange'>%s</font></h4>\" % mssg))\n",
        "    warns.append(mssg)\n",
        "\n",
        "display(Markdown(\"#### Checking Python version...\"))\n",
        "if sys.version_info < MIN_REQ_PYTHON:\n",
        "    setup_err(\"Python version\")\n",
        "    display(Markdown('Check the Kernel->Change Kernel menu and ensure that Python 3.10'))\n",
        "    display(Markdown('or later is selected as the active kernel.'))\n",
        "else:\n",
        "    setup_ok(\n",
        "        \"Python version {}.{}.{}\".format(\n",
        "            sys.version_info[0], sys.version_info[1], sys.version_info[2]\n",
        "        )\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Package Import Check\n",
        "\n",
        "This section checks the import of `msticpy` and its dependent packages.\n",
        "\n",
        "> **Note**\n",
        "> If you are repeatedly seeing packages going missing when working in Azure Notebooks\n",
        "> this may be because the docker containers running the Python kernel are\n",
        "> recycled after a few hours when not in use. This causes the environments\n",
        "> to reset to defaults.\n",
        "> \n",
        "> To prevent this you should configure you Azure Notebooks project with a\n",
        "> requirements.txt file that is automatically run (and packages installed)\n",
        "> when the contain is initialized.\n",
        ">\n",
        "> For details on how to do this see [AzureNotebooks-ConfigurePythonVersion](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/AzureNotebooks-ConfigurePythonVersion.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "gather": {
          "logged": 1618342008582
        }
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "#### Checking msticpy..."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<h4><font color='blue'>msticpy version 1.1.0 - OK</font></h4>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "msticpy version installed: 1.1.0 latest published: 1.0.0\n",
            "Latest version is installed.\n"
          ]
        }
      ],
      "source": [
        "import importlib\n",
        "import pkg_resources\n",
        "import sys\n",
        "import warnings\n",
        "from IPython.display import display, HTML, Markdown\n",
        "\n",
        "MSTICPY_REQ_VERSION = \"2.12.0\"\n",
        "display(Markdown(\"#### Checking msticpy...\"))\n",
        "warn_mssg = []\n",
        "err_mssg = []\n",
        "restart_req = False\n",
        "\n",
        "def _get_pkg_version(version):\n",
        "    if isinstance(version, str):\n",
        "        return pkg_resources.parse_version(version)\n",
        "    elif isinstance(version, tuple):\n",
        "        return pkg_resources.parse_version(\".\".join(str(ver) for ver in version))\n",
        "    raise TypeError(f\"Unparseable type version {version}\")\n",
        "\n",
        "MISSING_PKG_ERR = \"\"\"\n",
        "    <h3><font color='red'>Warning {package} is not installed or has an unsupported version</h3></font>\n",
        "    \"\"\"\n",
        "need_update = False\n",
        "try:\n",
        "    import msticpy\n",
        "    mp_version = _get_pkg_version(msticpy.__version__)\n",
        "    required_ver = _get_pkg_version(MSTICPY_REQ_VERSION)\n",
        "    if mp_version < required_ver:\n",
        "        setup_err(\n",
        "            f\"Installed version of msticpy is {mp_version}. \"\n",
        "            f\"msticpy {required_ver} or later is required.\"\n",
        "        )\n",
        "        need_update = True\n",
        "    else:\n",
        "        setup_ok(f\"msticpy version {msticpy.__version__}\")\n",
        "\n",
        "except ImportError:\n",
        "    display(HTML(MISSING_PKG_ERR.format(package=\"msticpy\")))\n",
        "    need_update = True\n",
        "\n",
        "\n",
        "if need_update:\n",
        "    resp = input(\"Install the package now? (y/n)\")\n",
        "    if resp.casefold().startswith(\"y\"):\n",
        "        %pip install --upgrade msticpy\n",
        "        if \"msticpy\" in sys.modules:\n",
        "            importlib.reload(sys.modules[\"msticpy\"])\n",
        "        else:\n",
        "            import msticpy\n",
        "        print(f\"msticpy installed - version {msticpy.__version__}\")\n",
        "\n",
        "    else:\n",
        "        setup_warn(\"msticpy missing or out-of-date.\")\n",
        "        display(Markdown(\"Please run `pip install --upgrade msticpy` to upgrade/install msticpy\"))\n",
        "\n",
        "try:\n",
        "    import msticpy\n",
        "    msticpy.check_version()\n",
        "except ImportError:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Pandas Version Check\n",
        "\n",
        "Many of the notebooks and msticpy features require a mininum\n",
        "pandas version of 0.25.0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "gather": {
          "logged": 1618342016369
        }
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "#### Checking pandas..."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<h4><font color='blue'>Pandas version 1.1.3 - OK</font></h4>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display(Markdown(\"#### Checking pandas...\"))\n",
        "PANDAS_REQ_VERSION = (1, 45, 0)\n",
        "need_update = False\n",
        "try:\n",
        "    import pandas as pd\n",
        "    pd_version = tuple([int(v) for v in pd.__version__.split(\".\")])\n",
        "    if pd_version < PANDAS_REQ_VERSION:\n",
        "        setup_err(\"pandas %s.%s.%s or later is required.\" % PANDAS_REQ_VERSION)\n",
        "        need_update = True\n",
        "except ImportError:\n",
        "    display(HTML(MISSING_PKG_ERR.format(package=\"pandas\")))\n",
        "    need_update = True\n",
        "else:\n",
        "    setup_ok(f\"Pandas version {pd.__version__}\")\n",
        "\n",
        "if need_update:\n",
        "    resp = input(\"Install the package now? (y/n)\")\n",
        "    if resp.casefold().startswith(\"y\"):\n",
        "        %pip install --upgrade pandas\n",
        "        if \"pandas\" in sys.modules:\n",
        "            importlib.reload(pd)\n",
        "        else:\n",
        "            import pandas as pd\n",
        "        print(f\"pandas installed - version {pd.__version__}\")\n",
        "\n",
        "    else:\n",
        "        setup_warn(\"pandas missing or out-of-date.\")\n",
        "        display(Markdown(\"Please run `pip install --upgrade pandas` to upgrade/install pandas\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Workspace Configuration Check\n",
        "\n",
        "This section checks for presence of configuration files `config.json`\n",
        "and `msticpyconfig.yaml`\n",
        "\n",
        "The `msticpyconfig.yaml` can store the workspace and tenant information\n",
        "for your Microsoft Sentinel workspace. It can also store values for multiple\n",
        "workspaces. If you have the values configured in this file you do not\n",
        "need to worry about the values in `config.json`.\n",
        "\n",
        "You can specify the location of your `msticpyconfig.yaml` in the\n",
        "environment variable `MSTICPYCONFIG`. This will make the file \n",
        "accessible to all notebooks running on the system. For\n",
        "more information on configuring `msticpyconfig.yaml` see the next\n",
        "cell [mstipcy Configuration](#msticpy-Configuration)\n",
        "\n",
        "If you want to transfer your workspace settings to `msticpyconfig.yaml`\n",
        "from `config.json`, simply copy the value of the `tenant_id` and \n",
        "`workspace_id` settings to the relevant section. \n",
        "\n",
        "> **Note** the value names in msticpyconfig.yaml use slightly different naming\n",
        "> conventions:\n",
        "```\n",
        "    WorkspaceId: 0cd830ff-60dc-40d1-8045-11d2b7b277e1\n",
        "    TenantId: aff2102d-1d6c-4501-9efb-6053ab7efb19\n",
        "```\n",
        "\n",
        "### Workspace Configuration - config.json\n",
        "Creating a Microsoft Notebooks project from Microsoft Sentinel \n",
        "will automatically create a `config.json` file in the root of\n",
        "your Azure Notebooks project and populate values\n",
        "for your Microsoft Sentinel workspace.\n",
        "\n",
        "If you have copied the notebooks elsewhere (e.g. to run them locally,\n",
        "or you are running them on a Data Science Virtual machine) you \n",
        "should copy this original config.json to the folder from which\n",
        "you are running notebooks.\n",
        "\n",
        "> **Note** if you are using a `msticpyconfig.yaml` to store your\n",
        "> workspace settings, most notebooks will take values from that.\n",
        "> As with `config.json` - you must have a locally accessible copy\n",
        "> of this file, so you will need to copy it to other systems if\n",
        "> you are running notebooks from there.\n",
        "\n",
        "\n",
        "If you are using the config.json (default config for Microsoft Sentinel\n",
        "with Azure Notebooks), your config.json should look something like this\n",
        "```json\n",
        "{\n",
        "  \"tenant_id\": \"aff2102d-1d6c-4501-9efb-6053ab7efb19\",\n",
        "  \"subscription_id\": \"9ce7caeb-1f42-4141-b076-7f448a00aceb\",\n",
        "  \"resource_group\": \"MyResourceGroup\",\n",
        "  \"workspace_id\": \"0cd830ff-60dc-40d1-8045-11d2b7b277e1\",\n",
        "  \"workspace_name\": \"MyResourceSubscription\"\n",
        "}\n",
        "```\n",
        "\n",
        "The tenant_id and workspace_id values must be configured, other values\n",
        "are optional but recommended."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "gather": {
          "logged": 1618342026006
        }
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "#### Checking Microsoft Sentinel Workspace config..."
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "No errors found.\n",
            "No warnings found.\n"
          ]
        },
        {
          "data": {
            "text/markdown": [
              "<h4><font color='blue'>Workspace configuration found in 'e:\\src\\microsoft\\msticpyconfig.yaml' - OK</font></h4>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "import json\n",
        "from pathlib import Path\n",
        "import uuid\n",
        "import yaml\n",
        "\n",
        "\n",
        "def valid_uuid(uuid_str):\n",
        "    try:\n",
        "        uuid.UUID(uuid_str)\n",
        "    except (ValueError, TypeError):\n",
        "        return False\n",
        "    return True\n",
        "\n",
        "def check_mp_config_ws(config_file):\n",
        "    with open(config_file, \"r\") as mp_yml:\n",
        "        mp_config = yaml.safe_load(mp_yml)\n",
        "    mp_errors = []\n",
        "    as_settings = mp_config.get(\"AzureSentinel\", {})\n",
        "    if not as_settings:\n",
        "        mp_errors.append(f\"Missing or empty 'AzureSentinel' section in {config_file}\")\n",
        "    ws_settings = as_settings.get(\"Workspaces\", {})\n",
        "    if not ws_settings:\n",
        "        mp_errors.append(f\"Missing or empty 'Workspaces' section in {config_file}\")\n",
        "    no_default = True\n",
        "    for ws, ws_settings in ws_settings.items():\n",
        "        if ws == \"Default\":\n",
        "            no_default = False\n",
        "        ws_id = ws_settings.get(\"WorkspaceId\")\n",
        "        if not ws_id and not valid_uuid(ws_id):\n",
        "            mp_errors.append(f\"Invalid GUID for WorkspaceId in {ws} section\")\n",
        "        ten_id = ws_settings.get(\"TenantId\")\n",
        "        if not ten_id and not valid_uuid(ten_id):\n",
        "            mp_errors.append(f\"Invalid GUID for TenantId in {ws} section\")\n",
        "    warnings = [\"No default workspace set\"] if no_default else []\n",
        "    return mp_errors, warnings\n",
        "\n",
        "\n",
        "try:\n",
        "    from msticpy.common.pkg_config import validate_config\n",
        "except ImportError:\n",
        "    # Fall back on local check if we cannot import from MP\n",
        "    validate_config = check_mp_config_ws\n",
        "\n",
        "\n",
        "def check_json_config(json_path):\n",
        "    j_conf_errs = []\n",
        "    with open(json_path, \"r\") as json_file:\n",
        "        conf_json = json.load(json_file)\n",
        "    conf_tenant = conf_json.get(\"tenant_id\")\n",
        "    if conf_tenant == \"{{cookiecutter.tenant_id}}\":\n",
        "        j_conf_errs.append(\"Tenant Id is set to default value\")\n",
        "    elif not valid_uuid(conf_tenant):\n",
        "        j_conf_errs.append(\"Tenant ID is not a valid GUID.\")\n",
        "    conf_ws = conf_json.get(\"workspace_id\")\n",
        "    if conf_ws == \"{{cookiecutter.workspace_id}}\":\n",
        "        j_conf_errs.append(\"Workspace Id is set to default value\")\n",
        "    elif not valid_uuid(conf_ws):\n",
        "        j_conf_errs.append(\"Workspace ID is not a valid GUID.\")\n",
        "    return j_conf_errs\n",
        "\n",
        "\n",
        "def get_aml_user_folder():\n",
        "    \"\"\"Return the root of the user folder.\"\"\"\n",
        "    user_path = Path(\"/\")\n",
        "    path_parts = Path(\".\").absolute().parts\n",
        "    for idx, part in enumerate(path_parts):\n",
        "        if part.casefold() == \"users\":\n",
        "            user_path = user_path.joinpath(part).joinpath(path_parts[idx + 1])\n",
        "            break\n",
        "        user_path = user_path.joinpath(part)\n",
        "    return user_path\n",
        "\n",
        "\n",
        "mp_warnings = []\n",
        "display(Markdown(\"#### Checking Microsoft Sentinel Workspace config...\"))\n",
        "mp_path = os.environ.get(\"MSTICPYCONFIG\", \"./msticpyconfig.yaml\")\n",
        "if not Path(mp_path).exists():\n",
        "    if Path(get_aml_user_folder()).joinpath(\"msticpyconfig.yaml\"):\n",
        "        setup_warn(\n",
        "            \"A 'msticpyconfig.yaml' was found in the root of your user folder\"\n",
        "            + \" but the MSTICPYCONFIG variable is not set. This file will be\"\n",
        "            + \" used by default.\"\n",
        "        )\n",
        "        mp_path = str(Path(get_aml_user_folder()).joinpath(\"msticpyconfig.yaml\"))\n",
        "if Path(mp_path).exists():\n",
        "    mp_errs, mp_warnings = validate_config(config_file=mp_path)\n",
        "else:\n",
        "    mp_errs = [f\"{mp_path} not found\"]\n",
        "\n",
        "DEF_CONF_JSON = \"./config.json\"\n",
        "jc_errs = []\n",
        "if Path(DEF_CONF_JSON).exists():\n",
        "    jc_errs = check_json_config(DEF_CONF_JSON)\n",
        "\n",
        "if jc_errs and mp_errs:\n",
        "    setup_err(\"No valid workspace configuration found.\")\n",
        "    if jc_errs:\n",
        "        print(jc_errs)\n",
        "    if mp_errs:\n",
        "        print(mp_errs)\n",
        "else:\n",
        "    if not jc_errs:\n",
        "        setup_ok(f\"Workspace configuration found in '{DEF_CONF_JSON}'\")\n",
        "    if not mp_errs:\n",
        "        setup_ok(f\"Workspace configuration found in '{mp_path}'\")\n",
        "    else:\n",
        "        setup_warn(f\"Workspace configuration: Cannot find msticpy config file {mp_path}\")\n",
        "    if mp_warnings:\n",
        "        display(Markdown(f\"<h5><font color='orange'>{', '.join(mp_warnings)}</font></h5>\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# msticpy Initialization\n",
        "\n",
        "This section duplicates the setup cells of most of the notebooks.\n",
        "It may duplicate warnings seen in the previous cell (since\n",
        "it runs some of the same checks).\n",
        "\n",
        "For more information on `msticpy` configuration file settings,\n",
        "please refer to the following items:\n",
        "\n",
        "- [Configuration guide notebook](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb)\n",
        "- [msticpy configuration documentation](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "gather": {
          "logged": 1618342035207
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<h3>Starting Notebook setup...</h3>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Note: you may need to scroll down this cell to see the full output."
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<h4>Starting notebook pre-checks...</h4>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Checking Python kernel version..."
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Recommended: switch to using the 'Python 3.8 - AzureML' notebook kernel if this is available."
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Info: Python kernel version 3.7.10 OK<br>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Checking msticpy version...<br>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Info: msticpy version 1.0.0 OK<br>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<h4>Notebook pre-checks complete.</h4>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "msticpy version installed: 1.1.0 latest published: 1.0.0\n",
            "Latest version is installed.\n",
            "Processing imports....\n",
            "pandas imported (alias=pd)\n",
            "get_ipython imported from IPython (alias=None)\n",
            "display imported from IPython.display (alias=None)\n",
            "HTML imported from IPython.display (alias=None)\n",
            "Markdown imported from IPython.display (alias=None)\n",
            "ipywidgets imported (alias=widgets)\n",
            "Path imported from pathlib (alias=None)\n",
            "matplotlib.pyplot imported (alias=plt)\n",
            "MatplotlibDeprecationWarning imported from matplotlib (alias=None)\n",
            "seaborn imported (alias=sns)\n",
            "numpy imported (alias=np)\n",
            "pandas imported version 1.1.3\n",
            "QueryProvider imported from msticpy.data (alias=None)\n",
            "FoliumMap imported from msticpy.nbtools.foliummap (alias=None)\n",
            "md imported from msticpy.common.utility (alias=None)\n",
            "md_warn imported from msticpy.common.utility (alias=None)\n",
            "WorkspaceConfig imported from msticpy.common.wsconfig (alias=None)\n",
            "Pivot imported from msticpy.datamodel.pivot (alias=None)\n",
            "entities imported from msticpy.datamodel (alias=None)\n",
            "All items imported from msticpy.nbtools\n",
            "All items imported from msticpy.sectools\n",
            "Imported: pd (pandas); IPython.get_ipython; IPython.display.display; IPython.display.HTML; IPython.display.Markdown; widgets (ipywidgets); pathlib.Path; plt (matplotlib.pyplot); matplotlib.MatplotlibDeprecationWarning; sns (seaborn); np (numpy); msticpy.data.QueryProvider; msticpy.nbtools.foliummap.FoliumMap; msticpy.common.utility.md; msticpy.common.utility.md_warn; msticpy.common.wsconfig.WorkspaceConfig; msticpy.datamodel.pivot.Pivot; msticpy.datamodel.entities\n",
            "Checking configuration....\n",
            "No errors found.\n",
            "No warnings found.\n",
            "Setting notebook options....\n",
            "Friendly exceptions enabled.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<h3>Notebook setup complete</h3>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<h4><font color='blue'>MSTICPy configuration ran without critical errors. - OK</font></h4>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from pathlib import Path\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "REQ_PYTHON_VER = \"3.10\"\n",
        "REQ_MSTICPY_VER = \"2.12.0\"\n",
        "REQ_MP_EXTRAS = []\n",
        "\n",
        "display(HTML(\"<h3>Starting Notebook setup...</h3>\"))\n",
        "# If not using Azure Notebooks, install msticpy with\n",
        "# %pip install msticpy\n",
        "\n",
        "import msticpy as mp\n",
        "nb_init_result = mp.init_notebook(\n",
        "    namespace=globals(),\n",
        "    verbose=True\n",
        ")\n",
        "if nb_init_result:\n",
        "    setup_ok(\"MSTICPy initialization ran without critical errors.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# msticpy general troubleshooting\n",
        "\n",
        "To help determine the cause of a problem you can turn on the following settings:\n",
        "\n",
        "1. MSTICPY debug logging\n",
        "2. Verbose reporting of Exceptions in Jupyter notebooks\n",
        "\n",
        "How to do these is covered in the code below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "# MSTICPy debug logging\n",
        "\n",
        "import msticpy as mp\n",
        "mp.set_logging_level(\"DEBUG\")\n",
        "\n",
        "# Then re-run the cell(s) that you are having problems with"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "From the logging you can see more detail of where an operation in the code\n",
        "is failing.\n",
        "\n",
        "```\n",
        "2024-06-17 13:27:23,906: INFO - Read 3 queries from E:\\src\\msticpy\\msticpy\\data\\queries\\m365d\\kql_m365_identity.yaml (data_query_reader#89)\n",
        "2024-06-17 13:27:23,910: INFO - Read 5 queries from E:\\src\\msticpy\\msticpy\\data\\queries\\m365d\\kql_m365_network.yaml (data_query_reader#89)\n",
        "2024-06-17 13:27:23,925: INFO - Read 4 queries from E:\\src\\msticpy\\msticpy\\data\\queries\\m365d\\kql_m365_process.yaml (data_query_reader#89)\n",
        "2024-06-17 13:27:23,946: INFO - Read 4 queries from E:\\src\\msticpy\\msticpy\\data\\queries\\m365d\\kql_m365_user.yaml (data_query_reader#89)\n",
        "2024-06-17 13:27:23,946: INFO - Adding query functions to provider (data_providers#137)\n",
        "2024-06-17 13:27:23,966: INFO - Initialization complete. (data_providers#140)\n",
        "2024-06-17 13:27:23,967: INFO - Calling connect on driver (data_providers#188)\n",
        "2024-06-17 13:27:23,968: INFO - WorkspaceConfig created from workspace name None (azure_monitor_driver#440)\n",
        "Attempting connection to Key Vault using cli credentials...\n",
        "2024-06-17 13:27:23,974: INFO - az_connect_core - using global cloud and endpoint: https://login.microsoftonline.com/ (azure_auth_core#290)\n",
        "2024-06-17 13:27:23,974: INFO - TenantId:  None, requested auth methods: cli (azure_auth_core#294)\n",
        "2024-06-17 13:27:23,975: INFO - Cred types added to chained credential: AzureCliCredential (azure_auth_core#382)\n",
        "done\n",
        "2024-06-17 13:27:29,407: INFO - az_connect_core - using global cloud and endpoint: https://login.microsoftonline.com/ (azure_auth_core#290)\n",
        "2024-06-17 13:27:29,407: INFO - TenantId:  72f988bf-86f1-41af-91ab-2d7cd011db47, requested auth methods: cli, interactive (azure_auth_core#294)\n",
        "2024-06-17 13:27:29,407: INFO - Cred types added to chained credential: AzureCliCredential, InteractiveBrowserCredential (azure_auth_core#382)\n",
        "2024-06-17 13:27:29,407: INFO - Created query client. Auth type: <class 'azure.identity._credentials.chained.ChainedTokenCredential'>, Url: https://api.loganalytics.io/v1, Proxies: None (azure_monitor_driver#400)\n",
        "2024-06-17 13:27:29,407: INFO - az_connect_core - using global cloud and endpoint: https://login.microsoftonline.com/ (azure_auth_core#290)\n",
        "2024-06-17 13:27:29,407: INFO - TenantId:  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, requested auth methods: cli, interactive (azure_auth_core#294)\n",
        "2024-06-17 13:27:29,407: INFO - Cred types added to chained credential: AzureCliCredential, InteractiveBrowserCredential (azure_auth_core#382)\n",
        "2024-06-17 13:27:33,668: INFO - Schema request to https://management.azure.com/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourcegroups/soc/providers/Microsoft.OperationalInsights/workspaces/CyberSecuritySOC/tables?api-version=2021-12-01-preview (azure_monitor_driver#607)\n",
        "2024-06-17 13:27:36,531: INFO - Schema retrieved from workspace. 723 tables found. (azure_monitor_driver#618)\n",
        "2024-06-17 13:27:36,547: INFO - Adding query pivot functions (data_providers#213)\n",
        "connected\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Exception reporting mode: Verbose\n"
          ]
        },
        {
          "ename": "ValueError",
          "evalue": "This is a deliberate error",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "Cell \u001b[1;32mIn[6], line 13\u001b[0m\n\u001b[0;32m     10\u001b[0m     my_param \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtest\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m     11\u001b[0m     func_with_error(param\u001b[38;5;241m=\u001b[39mmy_param)\n\u001b[1;32m---> 13\u001b[0m func_calling_error()\n",
            "Cell \u001b[1;32mIn[6], line 11\u001b[0m, in \u001b[0;36mfunc_calling_error\u001b[1;34m()\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mfunc_calling_error\u001b[39m():\n\u001b[0;32m     10\u001b[0m     my_param \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtest\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m---> 11\u001b[0m     func_with_error(param\u001b[38;5;241m=\u001b[39mmy_param)\n        my_param \u001b[1;34m= 'test'\u001b[0m\n",
            "Cell \u001b[1;32mIn[6], line 7\u001b[0m, in \u001b[0;36mfunc_with_error\u001b[1;34m(*args=(), **kwargs={'param': 'test'})\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mfunc_with_error\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m      6\u001b[0m     param \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mparam\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m----> 7\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThis is a deliberate error\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
            "\u001b[1;31mValueError\u001b[0m: This is a deliberate error"
          ]
        }
      ],
      "source": [
        "# Set verbose exception reporting\n",
        "\n",
        "%xmode Verbose\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Verbose Exception tracing\n",
        "\n",
        "Notice that in verbose mode you can see the values passed\n",
        "as parameters to functions, which helps a lot when diagnosing what has gone wrong\n",
        "\n",
        "#### Example code with a failure\n",
        "\n",
        "```python\n",
        "# Example\n",
        "def func_with_error(*args, **kwargs):\n",
        "    param = kwargs.get(\"param\")\n",
        "    raise ValueError(\"This is a deliberate error\")\n",
        "\n",
        "def func_calling_error():\n",
        "    my_param = \"test\"\n",
        "    func_with_error(param=my_param)\n",
        "\n",
        "func_calling_error()\n",
        "```\n",
        "\n",
        "#### Resultant traceback\n",
        "\n",
        "```python\n",
        "---------------------------------------------------------------------------\n",
        "ValueError                                Traceback (most recent call last)\n",
        "Cell In[6], line 13\n",
        "     10     my_param = \"test\"\n",
        "     11     func_with_error(param=my_param)\n",
        "---> 13 func_calling_error()\n",
        "\n",
        "Cell In[6], line 11\n",
        "      9 def func_calling_error():\n",
        "     10     my_param = \"test\"\n",
        "---> 11     func_with_error(param=my_param)\n",
        "        my_param = 'test'\n",
        "\n",
        "Cell In[6], line 7\n",
        "      5 def func_with_error(*args, **kwargs):\n",
        "      6     param = kwargs.get(\"param\")\n",
        "----> 7     raise ValueError(\"This is a deliberate error\")\n",
        "\n",
        "ValueError: This is a deliberate error\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Summary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "<h3><font color='blue'><u>Info/Success:</u></font>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<font color='blue'>Python version 3.7.10</font>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/markdown": [
              "<font color='blue'>MSTICPy configuration ran with errors.</font>"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "if errors:\n",
        "    display(Markdown(f\"<h3><font color='red'><u>{len(errors)} errors:</u></font>\"))\n",
        "    for mssg in errors:\n",
        "        display(Markdown(f\"<font color='red'>{mssg}</font>\"))\n",
        "if warns:\n",
        "    display(Markdown(f\"<h3><font color='orange'><u>{len(warns)} warnings:</u></font>\"))\n",
        "    for mssg in warns:\n",
        "        display(Markdown(f\"<font color='orange'>{mssg}</font>\"))\n",
        "display(Markdown(f\"<h3><font color='blue'><u>Info/Success:</u></font>\"))\n",
        "for mssg in info:\n",
        "    display(Markdown(f\"<font color='blue'>{mssg}</font>\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "hide_input": false,
    "kernel_info": {
      "name": "python310-sdkv2"
    },
    "kernelspec": {
      "display_name": "Python 3.10 - SDK v2",
      "language": "python",
      "name": "python310-sdkv2"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    },
    "microsoft": {
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      }
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": false,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "state": {
          "49464ef5c1984b9d94aa835d7377e7a8": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "1.2.0",
            "model_name": "LayoutModel",
            "state": {
              "width": "95%"
            }
          },
          "510b64fcbdfc4d75aad44b620a498f5e": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "1.2.0",
            "model_name": "LayoutModel",
            "state": {
              "width": "95%"
            }
          },
          "cf482f22059d43cbbc1eaffd642b5c7a": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "1.2.0",
            "model_name": "LayoutModel",
            "state": {
              "width": "95%"
            }
          },
          "dd8b84737f684f3c9b2da61bdcb0efa3": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "1.2.0",
            "model_name": "LayoutModel",
            "state": {
              "width": "95%"
            }
          },
          "f2a6bc6c00d14a6ab2affd8e742b47c3": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "1.2.0",
            "model_name": "LayoutModel",
            "state": {
              "width": "95%"
            }
          }
        },
        "version_major": 2,
        "version_minor": 0
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}