See open-metadata/OpenMetadata@9849564 from refs/heads/main

content/partials/v1.6/connectors/yaml/pipeline/source-config-def.md

@@ -4,14 +4,20 @@
 
 The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
 
-**dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it.
+- **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it.
 
-**includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion.
+- **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion.
 
-**includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true`
+- **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true`
 
-**markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
+- **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
 
-**pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude.
+- **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude.
+
+- **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`.
+
+- **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`.
+
+- **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`.
 
 {% /codeInfo %}

content/partials/v1.6/connectors/yaml/pipeline/source-config.md

@@ -16,4 +16,7 @@
       #   excludes:
       #     - pipeline3
       #     - pipeline4
+      # includeOwners: true # false
+      # overrideLineage: false # true
+      # overrideMetadata: false # true
 ```

content/partials/v1.7/connectors/yaml/pipeline/source-config-def.md

@@ -4,14 +4,20 @@
 
 The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json):
 
-**dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it.
+- **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it.
 
-**includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion.
+- **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion.
 
-**includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true`
+- **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true`
 
-**markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
+- **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system.
 
-**pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude.
+- **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude.
+
+- **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`.
+
+- **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`.
+
+- **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`.
 
 {% /codeInfo %}

content/partials/v1.7/connectors/yaml/pipeline/source-config.md

@@ -16,4 +16,7 @@
       #   excludes:
       #     - pipeline3
       #     - pipeline4
+      # includeOwners: true # false
+      # overrideLineage: false # true
+      # overrideMetadata: false # true
 ```

content/v1.6.x/connectors/pipeline/dagster/yaml.md

@@ -70,6 +70,10 @@ This is a sample config for Dagster:
 
 {% /codeInfo %}
 
+{% codeInfo srNumber=3 %}
+**timeout** : Connection Time Limit Between OM and Dagster Graphql API in second
+{% /codeInfo %}
+
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
@@ -96,6 +100,9 @@ source:
 ```yaml {% srNumber=2 %}
         token: token
 ```
+```yaml {% srNumber=3 %}
+        # timeout: 1000
+```
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 

content/v1.6.x/connectors/pipeline/flink/yaml.md

@@ -95,9 +95,18 @@ source:
 ```
 ```yaml {% srNumber=3 %}
         sslConfig:
-            caCertificate:
-            sslCertificate:
-            sslKey:
+                caCertificate: |
+                        -----BEGIN CERTIFICATE-----
+                        sample certificate
+                        -----END CERTIFICATE-----
+                sslCertificate: |
+                        -----BEGIN CERTIFICATE-----
+                        sample certificate
+                        -----END CERTIFICATE-----
+                sslKey: |
+                        -----BEGIN PRIVATE KEY-----
+                        sample certificate
+                        -----END PRIVATE KEY-----
 ```
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}

content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md

@@ -99,6 +99,22 @@ This is a sample config for Glue:
 
 {% /codeInfo %}
 
+{% codeInfo srNumber=6 %}
+**profileName**: The name of a profile to use with the boto session.
+{% /codeInfo %}
+
+{% codeInfo srNumber=7 %}
+**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
+{% /codeInfo %}
+
+{% codeInfo srNumber=8 %}
+**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
+{% /codeInfo %}
+
+{% codeInfo srNumber=9 %}
+**assumeRoleSourceIdentity**:  The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
+{% /codeInfo %}
+
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
@@ -134,6 +150,19 @@ source:
 ```yaml {% srNumber=5 %}
         # endPointURL: https://glue.us-east-2.amazonaws.com/
 ```
+```yaml {% srNumber=6 %}
+        # profileName: ""
+```
+```yaml {% srNumber=7 %}
+        # assumeRoleArn: ""
+```
+```yaml {% srNumber=8 %}
+        # assumeRoleSessionName: OpenMetadataSession
+```
+```yaml {% srNumber=9 %}
+        # assumeRoleSourceIdentity : ""
+```
+
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 

content/v1.6.x/connectors/pipeline/kafkaconnect/yaml.md

@@ -78,6 +78,9 @@ This is a sample config for KafkaConnect:
 
 {% /codeInfo %}
 
+{% codeInfo srNumber=4 %}
+**messagingServiceName**: Name of the Kafka Messaging Service associated with this KafkaConnect Pipeline Service. e.g. local_kafka.
+{% /codeInfo %}
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
@@ -109,6 +112,9 @@ source:
           username: username
           password: password
 ```
+```yaml {% srNumber=4 %}
+        # messagingServiceName: ""
+```
 
 {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 

content/v1.7.x-SNAPSHOT/connectors/pipeline/dagster/yaml.md

@@ -19,13 +19,13 @@ Configure and schedule Dagster metadata and profiler workflows from the OpenMeta
 - [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 
-{% partial file="/v1.7/connectors/external-ingestion-deployment.md" /%}
+{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
 
 ## Requirements
 
 ### Python Requirements
 
-{% partial file="/v1.7/connectors/python-requirements.md" /%}
+{% partial file="/v1.6/connectors/python-requirements.md" /%}
 
 To run the Dagster ingestion, you will need to install:
 
@@ -70,12 +70,16 @@ This is a sample config for Dagster:
 
 {% /codeInfo %}
 
+{% codeInfo srNumber=3 %}
+**timeout** : Connection Time Limit Between OM and Dagster Graphql API in second
+{% /codeInfo %}
+
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
 
 {% /codeInfoContainer %}
 
@@ -96,15 +100,18 @@ source:
 ```yaml {% srNumber=2 %}
         token: token
 ```
+```yaml {% srNumber=3 %}
+        # timeout: 1000
+```
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
 
 {% /codeBlock %}
 
 {% /codePreview %}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-cli.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}

content/v1.7.x-SNAPSHOT/connectors/pipeline/flink/yaml.md

@@ -19,13 +19,13 @@ Configure and schedule Flink metadata and profiler workflows from the OpenMetada
 - [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 
-{% partial file="/v1.7/connectors/external-ingestion-deployment.md" /%}
+{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
 
 ## Requirements
 
 ### Python Requirements
 
-{% partial file="/v1.7/connectors/python-requirements.md" /%}
+{% partial file="/v1.6/connectors/python-requirements.md" /%}
 
 To run the Flink ingestion, you will need to install:
 
@@ -68,11 +68,11 @@ This is a sample config for Flink:
 {% /codeInfo %}
 
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
 
 {% /codeInfoContainer %}
 
@@ -95,19 +95,28 @@ source:
 ```
 ```yaml {% srNumber=3 %}
         sslConfig:
-            caCertificate:
-            sslCertificate:
-            sslKey:
+                caCertificate: |
+                        -----BEGIN CERTIFICATE-----
+                        sample certificate
+                        -----END CERTIFICATE-----
+                sslCertificate: |
+                        -----BEGIN CERTIFICATE-----
+                        sample certificate
+                        -----END CERTIFICATE-----
+                sslKey: |
+                        -----BEGIN PRIVATE KEY-----
+                        sample certificate
+                        -----END PRIVATE KEY-----
 ```
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
 
 {% /codeBlock %}
 
 {% /codePreview %}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-cli.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}

content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md

@@ -17,7 +17,7 @@ Configure and schedule Glue metadata and profiler workflows from the OpenMetadat
 - [Requirements](#requirements)
 - [Metadata Ingestion](#metadata-ingestion)
 
-{% partial file="/v1.7/connectors/external-ingestion-deployment.md" /%}
+{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
 
 ## Requirements
 
@@ -32,7 +32,7 @@ The user must have the following permissions for the ingestion to run successful
 
 ### Python Requirements
 
-{% partial file="/v1.7/connectors/python-requirements.md" /%}
+{% partial file="/v1.6/connectors/python-requirements.md" /%}
 
 To run the Glue ingestion, you will need to install:
 
@@ -99,12 +99,28 @@ This is a sample config for Glue:
 
 {% /codeInfo %}
 
+{% codeInfo srNumber=6 %}
+**profileName**: The name of a profile to use with the boto session.
+{% /codeInfo %}
+
+{% codeInfo srNumber=7 %}
+**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
+{% /codeInfo %}
+
+{% codeInfo srNumber=8 %}
+**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
+{% /codeInfo %}
+
+{% codeInfo srNumber=9 %}
+**assumeRoleSourceIdentity**:  The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
+{% /codeInfo %}
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config-def.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
+
+{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
 
 {% /codeInfoContainer %}
 
@@ -134,15 +150,28 @@ source:
 ```yaml {% srNumber=5 %}
         # endPointURL: https://glue.us-east-2.amazonaws.com/
 ```
+```yaml {% srNumber=6 %}
+        # profileName: ""
+```
+```yaml {% srNumber=7 %}
+        # assumeRoleArn: ""
+```
+```yaml {% srNumber=8 %}
+        # assumeRoleSessionName: OpenMetadataSession
+```
+```yaml {% srNumber=9 %}
+        # assumeRoleSourceIdentity : ""
+```
+
 
-{% partial file="/v1.7/connectors/yaml/pipeline/source-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-sink.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
 
-{% partial file="/v1.7/connectors/yaml/workflow-config.md" /%}
+{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
 
 {% /codeBlock %}
 
 {% /codePreview %}
 
-{% partial file="/v1.7/connectors/yaml/ingestion-cli.md" /%}
+{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}