Merge pull request #43 from mandy-chessell/uc-demo

Overview notebook for workspaces

workspaces/cataloguing-and-surveys/apache-atlas/survey-and-catalog-atlas.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "96696a08-1a3c-4cc6-a9f4-1d79ee79087f",
+   "metadata": {},
+   "source": [
+    "<!-- SPDX-License-Identifier: CC-BY-4.0 -->\n",
+    "<!-- Copyright Contributors to the ODPi Egeria project 2024. -->\n",
+    "\n",
+    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "\n",
+    "### Egeria Workbook\n",
+    "\n",
+    "# Survey and Catalog Apache Atlas\n",
+    "\n",
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "caf08eaf-4f79-4c1a-9f87-ad4bf4af6fe0",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

workspaces/cataloguing-and-surveys/cataloguing-and-surveys.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "94577c57-e998-4a20-85b6-d4f0486f6655",
+   "metadata": {},
+   "source": [
+    "<!-- SPDX-License-Identifier: CC-BY-4.0 -->\n",
+    "<!-- Copyright Contributors to the ODPi Egeria project 2024. -->\n",
+    "\n",
+    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "\n",
+    "### Egeria Workbook\n",
+    "\n",
+    "# Cataloguing and surveys\n",
+    "\n",
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "559977df-32bc-4457-b025-00f62c8f9c6b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

workspaces/cataloguing-and-surveys/files/survey-and-catalog-files.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bbf73747-32b1-4088-9720-b5dcb7d5bbd1",
+   "metadata": {},
+   "source": [
+    "<!-- SPDX-License-Identifier: CC-BY-4.0 -->\n",
+    "<!-- Copyright Contributors to the ODPi Egeria project 2024. -->\n",
+    "\n",
+    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "\n",
+    "### Egeria Workbook\n",
+    "\n",
+    "# Cataloguing and surveying files\n",
+    "\n",
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "448b3b48-1ecf-49da-9445-117fdd9e24fb",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

workspaces/egeria-workbooks.ipynb

@@ -1,34 +1,126 @@
 {
  "cells": [
   {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "06211835-15de-44c6-b6f2-bd22f082f4ae",
+   "cell_type": "markdown",
+   "id": "a183522c-d414-4c6d-a957-e15d8ddf0f4b",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# SPDX-License-Identifier: Apache-2.0\n",
-    "# Copyright Contributors to the ODPi Egeria project."
+    "<!-- SPDX-License-Identifier: CC-BY-4.0 -->\n",
+    "<!-- Copyright Contributors to the ODPi Egeria project 2024. -->\n",
+    "\n",
+    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "\n",
+    "## Egeria Workspaces\n",
+    "\n",
+    "This JupyterLab environment was created by the *egeria-workspaces* docker compose scripts that provision JupyterLab with Egeria, Apache Kafka and other optional runtimes.  It is designed for individuals, or small teams, who want to work with Egeria in a python environment.\n",
+    "\n",
+    "### pyegeria\n",
+    "\n",
+    "The *pyegeria* package is a set of python libraries for working with Egeria.  It includes python functions, a command level interface (CLI) and monitoring/query widgets for interacting with Egeria’s runtime.  \n",
+    "\n",
+    "The **[working with pyegeria](pyegeria/working-with-pyegeria.ipynb)** notebook shows you how to ensure you have the latest level of pyegeria and how to use its basic functions.\n"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a183522c-d414-4c6d-a957-e15d8ddf0f4b",
+   "id": "2b62cfc4-dba1-4cf2-8256-2120e2aba12c",
    "metadata": {},
    "source": [
-    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "### runtime\n",
+    "\n",
+    "Running in the background are five Egeria servers configured to support your work.  The image below shows the servers along with a description of their purpose and when they last started up.  The display is one of *pyegeria*'s monitoring widgets and you can see it running by issuing the `hey_egeria_ops show servers status --full` command in a Terminal window of JupyterLab. (Use CTRL-C to exit the monitor when you are finished).\n",
+    "\n",
+    "![Active Servers](pyegeria/images/full-server-status.png)\n",
+    "\n",
+    "Most *pyegeria* functions call the `view-server` which then directs the request to the appropriate backend server."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b8615fa8-8a60-47a9-9ecc-c13400a4716e",
+   "metadata": {},
+   "source": [
+    "### active-metadata-store\n",
+    "\n",
+    "The `active-metadata-store` is the principal metadata server.  It controls access to the metadata repository where your metadata is stored.  For example, the image below shows the output of the `hey_egeria_cat show list-archives` command issued in a terminal window.  This is a paged query command so you use the space bar to page through the output and `q` to exit.\n",
+    "\n",
+    "The command is issued to the `view-server`, which then calls the `active-metadata-store` to retrieve details of all of the content packs that were detected and catalogued when the servers started up.\n",
+    "\n",
+    "![List Archives](pyegeria/images/list-archives.png)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8dfd7edb-cfcc-4423-86bf-b502b80f4d65",
+   "metadata": {},
+   "source": [
+    "### content packs\n",
+    "\n",
+    "Content packs are files containing open metadata elements that can be loaded into an open metadata repository.  They contain useful definitions for performing certain tasks.  The **[Loading Content Packs](loading-metadata/loading-content-packs.ipynb)** notebook gives more details and explains how to work with content packs.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b3556dc-d6b7-448c-aea1-31a65adaeb5b",
+   "metadata": {},
+   "source": [
+    "### integration-daemon\n",
+    "\n",
+    "The `integration-daemon` is the server responsible for automated cataloguing and synchronizing of metadata between Egeria and other technologies.  It was the *integration-daemon* that automatically catalogued the content packs shown above.  The image below is from the pyegeria monitoring widget displayed using the `hey_egeria_ops show integrations status` command.  It shows the integration connectors currently loaded and running in the `integration-daemon`.  This picture shows the integration connector status just after this JupyterLab environment is started for the first time.  They are waiting for work.  This will change as you use `pyegeria` to catalog new systems.\n",
+    "\n",
+    "![Integration Daemon Monitor](pyegeria/images/integration-daemon-monitor.png)\n",
+    "\n",
+    "This display shows the status of the [integration connectors](https://egeria-project.org/concepts/integration-connector/) running in the `integration-daemon` server.  Each integration connector is responsible for cataloguing/synchronizing metadata with a particular type of technology.  For example, the `ContentPacksCataloguer` is the integration connector responsible for detecting the content packs installed in this environment and adding their details to the metadata repository managed by `active-metadata-store`.\n",
+    "\n",
+    "The **[Cataloguing and Surveys](cataloguing-and-surveys/cataloguing-and-surveys.ipynb)** workbooks explain how to set up the integration connectors for specific types of technologies.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a84e3e5b-727b-4b81-8110-42a7bc69cf5f",
+   "metadata": {},
+   "source": [
+    "### engine-host\n",
+    "\n",
+    "The `engine-host` server runs [governance actions](https://egeria-project.org/concepts/governance-action/).  These actions can validate or enrich metadata, issue actions to stewards, raise incident reports, configure the integration connectors in the `integration-daemon` and run surveys.   Governance actions issue requests to [Governance Engines](https://egeria-project.org/concepts/governance-engine/). The types of requests available in the `engine-host` server can be monitored using the `hey_egeria_ops show engines status`.  The image below shows the engines and their request types just after this JupyterLab environment is started for the first time.  \n",
+    "\n",
+    "![Engine Host Status](pyegeria/images/engine-host-status.png)\n",
+    "\n",
+    "The governance engines in `ASSIGNED` status are waiting for additional configuration which is found in the content packs.  As you start to work with Egeria, the status of the `engine-host` will change."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49af1711-0bfa-4696-9292-ecc5dd94d8fa",
+   "metadata": {},
+   "source": [
+    "### simple-metadata-store\n",
+    "\n",
+    "The `simple-metadata-store` is not typically used.  It is included in this environment to allow you to experiment with [Open Metadata Repository Cohorts](https://egeria-project.org/concepts/cohort-member/).  This is described in the **[working-with-cohorts](working-with-cohorts/working-with-cohorts.ipynb)** notebook.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0e31bc51-9a99-4124-957f-9431052d4bee",
+   "metadata": {},
+   "source": [
+    "### where next?\n",
     "\n",
-    "### Egeria Workspaces\n",
+    "If you are new to Egeria, we suggest:\n",
     "\n",
-    "This JupyterLab environment was created by the *egeria-workspaces* docker compose script that provisions JupyterLab with Egeria and Apache Kafka.  It is designed for individuals, or small teams, who want to work with Egeria in a python environment.\n",
+    "1. Walk through the **[Working with pyegeria](pyegeria/working-with-pyegeria.ipynb)** notebook to ensure you have the latest version running in your environment.\n",
+    "2. Try surveying and cataloguing files using the **[Survey and cataloguing files](cataloguing-and-surveys/files/survey-and-catalog-files.ipynb)**.\n",
+    "3. See how to query the wide range of metadata managed by Egeria with the **[Querying metadata](querying-metadata/querying-metadata.ipynb)** notebook.\n",
     "\n",
-    "The Egeria part of this includes pyegeria and a preconfigured runtime. pyegeria has  python functions, a CLI and monitoring/display widgets for interacting with Egeria’s runtime."
+    "If you have any questions, please feel free to ask questions and discuss topics of interests with [the Egeria community](https://egeria-project.org/guides/community/#asynchronous-dialog)."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9c418607-e889-4e4e-b1b8-b3e1f2d30cc2",
+   "id": "7f21c757-6656-43d2-96d2-05615112600b",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -50,7 +142,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.9"
+   "version": "3.11.10"
   }
  },
  "nbformat": 4,

workspaces/governance-actions/governance-actions.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "10ddf845-b572-4e50-82aa-49697b74c7c2",
+   "metadata": {},
+   "source": [
+    "<!-- SPDX-License-Identifier: CC-BY-4.0 -->\n",
+    "<!-- Copyright Contributors to the ODPi Egeria project 2024. -->\n",
+    "\n",
+    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
+    "\n",
+    "### Egeria Workbook\n",
+    "\n",
+    "# Governance actions\n",
+    "\n",
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "87558c8f-281d-48b5-9a40-f2ea9d7e4d66",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

workspaces/loading-metadata/loading-content-packs.ipynb

@@ -121,7 +121,7 @@
    "source": [
     "----\n",
     "\n",
-    "In the code below, the */deployments/content-packs/CocoComboArchive.omarchive* archive is loaded into the *active-metadata-server*.  This provides lots of metadata describing the Coco Pharmceuticals people and operations.  It does not matter if the same archive is loaded multiple times.  This does not result in duplicate versons of the same element.\n",
+    "In the code below, the */deployments/content-packs/CocoComboArchive.omarchive* archive is loaded into the *active-metadata-server*.  This provides lots of metadata describing the Coco Pharmceuticals people and operations.  It does not matter if the same archive is loaded multiple times.  This does not result in duplicate versions of the same element.\n",
     "\n",
     "-----"
    ]