Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add updates to workbooks #47

Merged
merged 1 commit into from
Oct 14, 2024
Merged

Add updates to workbooks #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion workspaces/cataloguing-and-surveys/cataloguing-and-surveys.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,30 @@
"\n",
"# Cataloguing and surveys\n",
"\n",
"## Introduction"
"## Introduction\n",
"\n",
"Organizations typically have a lot of deployed technology containing data and its associated metadata. Often you want to capture the metadata about these technologies. Egeria offers two options that can be used independently, but have extra value when they are used together.\n",
"\n",
"* A *Survey* - In this option, a survey service connects to the deployed technology, extracts the metadata and compiles a report of this metadata to allow you to view what is available. The report is stored in the open metadata repositories. Optionally this can be printed out in a markdown report, or a summay exported as a CSV file.\n",
"* Then you can *Catalog* the deployed technology - In this option, a cataloguing service connectors to the deployed technology, extracts the metadata and builds an open metadata catalog of the contents. This can be searched and enriched using Egeria's APIs and governance processes.\n",
"\n",
"Often the cataloguing service can be configured to only catalog requested resources (or it can be configured to ignore specific resources), allowing you to create a catalog of the most interesting/valuable resouces without cluttering up your catalog with irrelevant resources. The output from the survey can be used to decide what to catalog.\n",
"\n",
"Similarly, once you have your resources catalogued, a regular survey can be used to plot how the resource is changing over time.\n",
"\n",
"Egeria has pre-build services for popular open source technologies. For example:\n",
"\n",
"* [Cataloguing and surveying files on a file system](files/survey-and-catalog-files.ipynb)\n",
"* [Cataloguing and Surveying Unity Catalog servers](unity-catalog/survey-and-catalog-uc.ipynb)\n",
"* [Cataloguing and Surveying PostgreSQL Servers](postgres/survey-and-catalog-postgres.ipynb)\n",
"* [Cataloguing and Surveying Apache Atlas Servers](apache-atlas/survey-and-catalog-atlas.ipynb)\n",
"\n",
"In addition, it is possible to capture and distribute Open Lineage events:\n",
"\n",
"* [Publishing Open Lineage Events to Marquez](marquez/publish-to-marquez.ipynb)\n",
"* [Capture Open Lineage Event from an Apache Kafka Topic](apache-kafka/kafka-open-lineage-events.ipynb)\n",
"\n",
"----"
]
},
{
Expand Down
Loading