feat(ingestion): business-glossary - Add values and relatedTerms supp…

…ort (datahub-project#6148) * WIP * updated snapshot * merge * while creating BusinessGlossaryConfig object, the version field always get set to None * WIP * lint fix * remove field * doc update * rename the fields * review comments * lintFix * Fix the business glossary example link * small fixes to examples and docs * some small tweaks to the model annotations Co-authored-by: MohdSiddique Bagwan <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Gabe Lyons <[email protected]>
gabe-lyons · Nov 2, 2022 · 2c28b87 · 2c28b87
1 parent 2be400c
commit 2c28b87
Show file tree

Hide file tree

Showing 13 changed files with 411 additions and 29 deletions.
diff --git a/metadata-ingestion/docs/sources/business-glossary/datahub-business-glossary.md b/metadata-ingestion/docs/sources/business-glossary/datahub-business-glossary.md
@@ -16,6 +16,7 @@ The business glossary source file should be a `.yml` file with the following top
 **GlossaryNode**: a container of **GlossaryNode** and **GlossaryTerm** objects
 - **name**: name of the node
 - **description**: description of the node
+- **id**: (optional) identifier of the node (normally inferred from the name, see `enable_auto_id` config. Use this if you need a stable identifier)
 - **owners**: (optional) owners contains two nested fields
   - **users**: (optional) a list of user ids
   - **groups**: (optional) a list of group ids
@@ -25,6 +26,7 @@ The business glossary source file should be a `.yml` file with the following top
 **GlossaryTerm**: a term in your business glossary
 - **name**: name of the term
 - **description**: description of the term
+- **id**: (optional) identifier of the term (normally inferred from the name, see `enable_auto_id` config. Use this if you need a stable identifier)
 - **owners**: (optional) owners contains two nested fields
   - **users**: (optional) a list of user ids
   - **groups**: (optional) a list of group ids
@@ -35,7 +37,7 @@ The business glossary source file should be a `.yml` file with the following top
 - **contains**: (optional) List of **GlossaryTerm** that this term contains
 - **custom_properties**: A map of key/value pairs of arbitrary custom properties
 
-You can also view an example business glossary file checked in [here](../examples/bootstrap_data/business_glossary.yml)
+You can also view an example business glossary file checked in [here](../../../examples/bootstrap_data/business_glossary.yml)
 
 ## Compatibility
 

diff --git a/metadata-ingestion/docs/sources/business-glossary/datahub-business-glossary_recipe.yml b/metadata-ingestion/docs/sources/business-glossary/datahub-business-glossary_recipe.yml
@@ -3,6 +3,6 @@ source:
   config:
     # Coordinates
     file: /path/to/business_glossary_yaml
+    enable_auto_id: true # recommended to set to true so datahub will auto-generate guids from your term names
 
-sink:
-  # sink configs
+# sink configs if needed
diff --git a/metadata-ingestion/examples/bootstrap_data/business_glossary.yml b/metadata-ingestion/examples/bootstrap_data/business_glossary.yml
@@ -27,6 +27,8 @@ nodes:
         - mjames
     terms:
       - name: Email
+        ## An example of using an id to pin a term to a specific guid
+        # id: "urn:li:glossaryTerm:41516e310acbfd9076fffc2c98d2d1a3"
         description: An individual's email address
         inherits:
           - Classification.Confidential
@@ -59,3 +61,52 @@ nodes:
         term_source: "EXTERNAL"
         source_ref: FIBO
         source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Balance"
+  - name: Housing
+    description: Provides terms related to the housing domain
+    owners:
+      groups:
+        - interior
+    nodes:
+      - name: Colors
+        description: "Colors that are used in Housing construction"
+        terms:
+          - name: Red
+            description: "red color"
+            term_source: "EXTERNAL"
+            source_ref: FIBO
+            source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+
+          - name: Green
+            description: "green color"
+            term_source: "EXTERNAL"
+            source_ref: FIBO
+            source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+
+          - name: Pink
+            description: pink color
+            term_source: "EXTERNAL"
+            source_ref: FIBO
+            source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+    terms:
+      - name: WindowColor
+        description: Supported window colors
+        term_source: "EXTERNAL"
+        source_ref: FIBO
+        source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+        values:
+          - House.Colors.Red
+          - House.Colors.Pink
+
+      - name: Kitchen
+        description: a room or area where food is prepared and cooked.
+        term_source: "EXTERNAL"
+        source_ref: FIBO
+        source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+
+      - name: Spoon
+        description: an implement consisting of a small, shallow oval or round bowl on a long handle, used for eating, stirring, and serving food.
+        term_source: "EXTERNAL"
+        source_ref: FIBO
+        source_url: "https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Account"
+        related_terms:
+          - House.Kitchen
diff --git a/metadata-ingestion/examples/recipes/business_glossary_to_datahub.dhub.yaml b/metadata-ingestion/examples/recipes/business_glossary_to_datahub.dhub.yaml
@@ -2,10 +2,9 @@ source:
   type: datahub-business-glossary
   config:
     file: ./examples/bootstrap_data/business_glossary.yml
-
+    enable_auto_id: True
 
 sink:
-  type: datahub-rest 
+  type: datahub-rest
   config:
     server: http://localhost:8080
-