Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OTel trace group processor to doc website. #3236

Merged
merged 40 commits into from
Apr 25, 2023
Merged
Changes from 8 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
223447d
Add otel trace group processor.
carolxob Mar 6, 2023
3cb6241
Copy edits to improve table formatting.
carolxob Mar 8, 2023
64ff76e
Incorporated doc review feedback.
carolxob Mar 8, 2023
ccc9f2a
Minor updates.
carolxob Mar 21, 2023
31a228e
Minor updates.
carolxob Mar 21, 2023
1003c9c
Minor updates.
carolxob Mar 21, 2023
66fb914
Minor updates.
carolxob Mar 21, 2023
834e5a9
Tech review updates.
carolxob Mar 24, 2023
fc66c9f
Content modifications based on tech feedback.
carolxob Mar 24, 2023
644cde6
Content updates.
carolxob Mar 24, 2023
2fb5c0f
Removed unnecessary link.
carolxob Mar 24, 2023
e27644b
Removed markdown comments.
carolxob Mar 24, 2023
6402161
Minor edits.
carolxob Apr 13, 2023
ed15d5f
Changed processor name references to remove tic marks.
carolxob Apr 14, 2023
5419fee
Minor edits.
carolxob Apr 14, 2023
daf7467
Minor edits to fix links.
carolxob Apr 14, 2023
4ed82b6
Removed common metrics from Metrics section and updated default value…
carolxob Apr 18, 2023
d0f7b23
Minor edits.
carolxob Apr 19, 2023
c59d4ea
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
925ba5a
Updated OpenSearch security link.
carolxob Apr 24, 2023
449e7dc
Removed unnecessary column in Metrics table.
carolxob Apr 24, 2023
38c80a4
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
7e39698
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
6b4c774
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
3a28352
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
2fb50ab
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
c0df326
Minor update to title for accuracy.
carolxob Apr 24, 2023
145e3f0
Removed optional from each item in the Configuration table. All items…
carolxob Apr 24, 2023
c351ce5
Updated links for OS security and service.
carolxob Apr 25, 2023
a3d76f4
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ee89cfc
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
a6b24a0
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
787291f
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
e309116
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ffe1c0d
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ced0156
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ab1e483
Updates made based on editorial feedback.
carolxob Apr 25, 2023
8e4d69f
Minor update to phrasing.
carolxob Apr 25, 2023
b990e73
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
e0cc5bd
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
layout: default
title: OTel trace group
carolxob marked this conversation as resolved.
Show resolved Hide resolved
parent: Processors
grand_parent: Pipelines
nav_order: 45
---

# OTel trace group

The `OTel trace group` processor completes missing trace group related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/834f28fdf1df6d42a6666e91e6407474b88e7ec6/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend. The `OTel trace group` processor identifies the missing trace group information for a `spanId` by looking up the relevant fields in its root `span` stored in OpenSearch.

## OpenSearch

See the following example configuration for `OTel trace group` processor when using self-managed OpenSearch:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

<!--- Is this a YAML file?--->

```
pipeline:
...
processor:
- otel_trace_group:
hosts: ["https://localhost:9200"]
cert: path/to/cert
username: YOUR_USERNAME_HERE
password: YOUR_PASSWORD_HERE
```

See [OpenSearch security](https://github.com/opensearch-project/data-prepper/blob/834f28fdf1df6d42a6666e91e6407474b88e7ec6/data-prepper-plugins/opensearch/opensearch_security.md#L4) for a more detailed explanation of which OpenSearch credentials and permissions are required and how to configure those credentials for the `OTel trace group` processor.

## Amazon OpenSearch Service

See the following example configuration for the `OTel trace group` processor when using Amazon OpenSearch Service:

<!--- Is this a YAML file?--->

```
pipeline:
...
processor:
- otel_trace_group:
hosts: ["https://your-amazon-opensearch-service-endpoint"]
aws_sigv4: true
cert: path/to/cert
insecure: false
```

See [OpenSearch Sink Security](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/security.md) for a detailed explanation.

<!--- Does this security section need to be here? --->

## Configuration

You can configure the Amazon OpenSearch Service with the following options.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

<!--- Are we talking about the OTel processor or OpenSearch? This is confusing. --->

| Name | Required | Description | Default value
| -----| ----| -----------| -----|
| `hosts`| Yes | A list of IP addresses of OpenSearch nodes. | Default value |
| `cert` | No | A certificate authority (CA) certificate that is PEM-encoded. Accepts both .pem or .crt. This enables the client to trust the CA that has signed the certificate that OpenSearch is using. | `null` |
| `aws_sigv4` | No | A boolean flag to sign the HTTP request with AWS credentials. Only applies to Amazon OpenSearch Service. See [OpenSearch security](https://github.com/opensearch-project/data-prepper/blob/129524227779ee35a327c27c3098d550d7256df1/data-prepper-plugins/opensearch/security.md) for details. | `false`. |
| `aws_region` | No | A String that represents the region of Amazon OpenSearch Service domain, for example, `us-west-2`. Only applies to Amazon OpenSearch Service. | `us-east-1` |
| `aws_sts_role_arn`| No | An identity and access management (IAM) role that the sink plugin assumes to sign the request to the Amazon OpenSearch Service. If not provided, the plugin uses the default credentials. | `null` |
| `aws_sts_header_overrides` | No | A map of header overrides to make when assuming the IAM role for the sink plugin. | `null` |
| `insecure` | No | A boolean flag to turn off SSL certificate verification. If set to `true`, CA certificate verification is turned off and insecure HTTP requests are sent. | `false` |
| `username` | No | A string that contains the username and is used in the [internal users](https://opensearch.org/docs/latest/security/access-control/users-roles/) of your OpenSearch cluster. | `null` |
| `password` | No | A string that contains the password and is used in the [internal users](https://opensearch.org/docs/latest/security/access-control/users-roles/) of your OpenSearch cluster. | `null` |

## Metrics

The following table describes metrics that are common to all processors.

| Metric name | Type | Description |
| ------------- | ---- | -----------|
| `recordsIn` | Counter | Metric representing the ingress of records to a pipeline component. |
| `recordsOut` | Counter | Metric representing the egress of records from a pipeline component. |
| `timeElapsed` | Timer | Metric representing the time elapsed during execution of a pipeline component. |

The `otel_trace_group` processor includes the following custom counter metrics.

| Metric name | Type | Description |
| ------------- | -----------|
| `recordsInMissingTraceGroup` | Counter | The number of ingress records missing trace group fields. |
| `recordsOutFixedTraceGroup` | Counter | The number of egress records with successfully completed trace group fields. |
| `recordsOutMissingTraceGroup` | Counter | The number of egress records missing trace group fields. |