Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OTel trace group processor to doc website. #3236

Merged
merged 40 commits into from
Apr 25, 2023
Merged
Changes from 38 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
223447d
Add otel trace group processor.
carolxob Mar 6, 2023
3cb6241
Copy edits to improve table formatting.
carolxob Mar 8, 2023
64ff76e
Incorporated doc review feedback.
carolxob Mar 8, 2023
ccc9f2a
Minor updates.
carolxob Mar 21, 2023
31a228e
Minor updates.
carolxob Mar 21, 2023
1003c9c
Minor updates.
carolxob Mar 21, 2023
66fb914
Minor updates.
carolxob Mar 21, 2023
834e5a9
Tech review updates.
carolxob Mar 24, 2023
fc66c9f
Content modifications based on tech feedback.
carolxob Mar 24, 2023
644cde6
Content updates.
carolxob Mar 24, 2023
2fb5c0f
Removed unnecessary link.
carolxob Mar 24, 2023
e27644b
Removed markdown comments.
carolxob Mar 24, 2023
6402161
Minor edits.
carolxob Apr 13, 2023
ed15d5f
Changed processor name references to remove tic marks.
carolxob Apr 14, 2023
5419fee
Minor edits.
carolxob Apr 14, 2023
daf7467
Minor edits to fix links.
carolxob Apr 14, 2023
4ed82b6
Removed common metrics from Metrics section and updated default value…
carolxob Apr 18, 2023
d0f7b23
Minor edits.
carolxob Apr 19, 2023
c59d4ea
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
925ba5a
Updated OpenSearch security link.
carolxob Apr 24, 2023
449e7dc
Removed unnecessary column in Metrics table.
carolxob Apr 24, 2023
38c80a4
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
7e39698
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
6b4c774
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
3a28352
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
2fb50ab
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 24, 2023
c0df326
Minor update to title for accuracy.
carolxob Apr 24, 2023
145e3f0
Removed optional from each item in the Configuration table. All items…
carolxob Apr 24, 2023
c351ce5
Updated links for OS security and service.
carolxob Apr 25, 2023
a3d76f4
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ee89cfc
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
a6b24a0
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
787291f
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
e309116
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ffe1c0d
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ced0156
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
ab1e483
Updates made based on editorial feedback.
carolxob Apr 25, 2023
8e4d69f
Minor update to phrasing.
carolxob Apr 25, 2023
b990e73
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
e0cc5bd
Update _data-prepper/pipelines/configuration/processors/otel-trace-gr…
carolxob Apr 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
layout: default
title: otel_trace_group processor
parent: Processors
grand_parent: Pipelines
nav_order: 45
---

# otel_trace_group processor

The `otel_trace_group` processor completes missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/834f28fdf1df6d42a6666e91e6407474b88e7ec6/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend. The `otel_trace_group` processor identifies the missing trace group information for a `spanId` by looking up the relevant fields in its root `span` stored in OpenSearch.

## OpenSearch

When you connect to an OpenSearch cluster using your username and password, use the following `pipeline.yaml` file example to configure the `otel_trace_group` processor:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

``` YAML
pipeline:
...
processor:
- otel_trace_group:
hosts: ["https://localhost:9200"]
cert: path/to/cert
username: YOUR_USERNAME_HERE
password: YOUR_PASSWORD_HERE
```

See [OpenSearch security]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sinks/opensearch/#opensearch-cluster-security) for a more detailed explanation of which OpenSearch credentials and permissions are required and how to configure those credentials for the OTel trace group processor.

### Amazon OpenSearch Service

When you use [Amazon OpenSearch Service]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sinks/opensearch/#amazon-opensearch-service-domain-security), use the following `pipeline.yaml` file example to configure the `otel_trace_group` processor:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

``` YAML
pipeline:
...
processor:
- otel_trace_group:
hosts: ["https://your-amazon-opensearch-service-endpoint"]
aws_sigv4: true
cert: path/to/cert
insecure: false
```

## Configuration

You can configure the `otel_trace_group` processor with the following options.

| Name | Description | Default value |
| -----| ----| -----------|
| `hosts`| A list of IP addresses of OpenSearch nodes. Required. | No default value. |
| `cert` | A certificate authority (CA) certificate that is PEM encoded. Accepts both .pem or .crt. This enables the client to trust the CA that has signed the certificate that OpenSearch is using. | `null` |
| `aws_sigv4` | A Boolean flag used to sign the HTTP request with AWS credentials. Only applies to Amazon OpenSearch Service. See [OpenSearch security](https://github.com/opensearch-project/data-prepper/blob/129524227779ee35a327c27c3098d550d7256df1/data-prepper-plugins/opensearch/security.md) for details. | `false`. |
| `aws_region` | A string that represents the AWS Region of the Amazon OpenSearch Service domain, for example, `us-west-2`. Only applies to Amazon OpenSearch Service. | `us-east-1` |
| `aws_sts_role_arn`| An AWS Identity and Access Management (IAM) role that the sink plugin assumes to sign the request to Amazon OpenSearch Service. If not provided, the plugin uses the [default credentials](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html). | `null` |
| `aws_sts_header_overrides` | A map of header overrides that the IAM role assumes for the sink plugin. | `null` |
| `insecure` | A Boolean flag used to turn off SSL certificate verification. If set to `true`, CA certificate verification is turned off and insecure HTTP requests are sent. | `false` |
| `username` | A string that contains the username and is used in the [internal users](https://opensearch.org/docs/latest/security/access-control/users-roles/) `YAML` configuration file of your OpenSearch cluster. | `null` |
| `password` | A string that contains the password and is used in the [internal users](https://opensearch.org/docs/latest/security/access-control/users-roles/) `YAML` configuration file of your OpenSearch cluster. | `null` |

## Configuration option examples

You can define the configuration option values in the `aws_sts_header_overrides` option. See the following example:

```
aws_sts_header_overrides:
x-my-custom-header-1: my-custom-value-1
x-my-custom-header-2: my-custom-value-2
```

## Metrics

The following table describes custom metrics specific to the `otel_trace_group` processor.

| Metric name | Type | Description |
| ------------- | ---- | ----------- |
| `recordsInMissingTraceGroup` | Counter | The number of ingress records missing trace group fields. |
| `recordsOutFixedTraceGroup` | Counter | The number of egress records with successfully completed trace group fields. |
| `recordsOutMissingTraceGroup` | Counter | The number of egress records missing trace group fields. |