Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Fix rename processor 'otel_trace' to 'otel_traces' #5423

Merged
merged 20 commits into from
Nov 7, 2023
Merged

[DOC] Fix rename processor 'otel_trace' to 'otel_traces' #5423

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1ce4df0
Fix APIs File
samipak458 Oct 6, 2023
185c4a5
Merge pull request #1 from samipak458/patch01
samipak458 Oct 6, 2023
fbf0960
Merge branch 'opensearch-project:main' into main
samipak458 Oct 13, 2023
f615491
Update API files according to API template
samipak458 Oct 13, 2023
502fb25
Update exec-stored-script.md
Naarcha-AWS Oct 16, 2023
9543661
Merge branch 'opensearch-project:main' into main
samipak458 Oct 17, 2023
d394a06
Update cat-aliases.md
samipak458 Oct 17, 2023
6bb61fa
Update cat-allocation.md
samipak458 Oct 17, 2023
4b4675d
Update cat-cluster_manager.md
samipak458 Oct 17, 2023
fc8a276
Update cat-count.md
samipak458 Oct 17, 2023
2cfa11b
Update cat-field-data.md
samipak458 Oct 17, 2023
d5be4dd
Update cat-health.md
samipak458 Oct 17, 2023
73f6fe8
updated
samipak458 Oct 18, 2023
c931acd
updated
samipak458 Oct 18, 2023
adc0836
updated
samipak458 Oct 19, 2023
3e52be2
Merge remote-tracking branch 'upstream/main'
samipak458 Oct 19, 2023
b21a9d3
Merge remote-tracking branch 'upstream/main'
samipak458 Oct 19, 2023
37656af
Merge branch 'opensearch-project:main' into main
samipak458 Oct 28, 2023
e673d49
data prepper update
samipak458 Oct 28, 2023
2c51182
Update _data-prepper/managing-data-prepper/peer-forwarder.md
Naarcha-AWS Nov 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions _data-prepper/common-use-cases/trace-analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@

There are three processors for the trace analytics feature:

* *otel_trace_raw* - The *otel_trace_raw* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace/), and performs stateful processing, extraction, and completion of trace-group-related fields.
* *otel_trace_group* - The *otel_trace_group* processor fills in the missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/298e7931aa3b26130048ac3bde260e066857df54/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend.
* *otel_traces_raw* - The *otel_traces_raw* processor receives a collection of [span](https://github.com/opensearch-project/data-prepper/blob/fa65e9efb3f8d6a404a1ab1875f21ce85e5c5a6d/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records from [*otel-trace-source*]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace/), and performs stateful processing, extraction, and completion of trace-group-related fields.

Check failure on line 41 in _data-prepper/common-use-cases/trace-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'OTel' instead of 'otel'.

Raw Output:
{"message": "[Vale.Terms] Use 'OTel' instead of 'otel'.", "location": {"path": "_data-prepper/common-use-cases/trace-analytics.md", "range": {"start": {"line": 41, "column": 4}}}, "severity": "ERROR"}
samipak458 marked this conversation as resolved.
Show resolved Hide resolved
* *otel_traces_group* - The *otel_traces_group* processor fills in the missing trace-group-related fields in the collection of [span](https://github.com/opensearch-project/data-prepper/blob/298e7931aa3b26130048ac3bde260e066857df54/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the OpenSearch backend.
* *service_map_stateful* – The *service_map_stateful* processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards.


Expand All @@ -49,7 +49,7 @@

The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics:

* *otel-v1-apm-span* – The *otel-v1-apm-span* index stores the output from the [otel_trace_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
* *otel-v1-apm-span* – The *otel-v1-apm-span* index stores the output from the [otel_traces_raw]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/otel-trace-raw/) processor.
* *otel-v1-apm-service-map* – The *otel-v1-apm-service-map* index stores the output from the [service_map_stateful]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/service-map-stateful/) processor.

## Trace tuning
Expand Down Expand Up @@ -114,13 +114,13 @@

The following example demonstrates how to build a pipeline that supports the [OpenSearch Dashboards Observability plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines serve two different purposes and write to different OpenSearch indexes. The first pipeline prepares trace data for OpenSearch and enriches and ingests the span documents into a span index within OpenSearch. The second pipeline aggregates traces into a service map and writes service map documents into a service map index within OpenSearch.

Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_trace_raw_prepper` processor. The `otel_trace_raw` processor replaces the `otel_trace_raw_prepper` processor and supports some of Data Prepper's recent data model changes. Instead, you should use the `otel_trace_raw` processor. See the following YAML file example:
Starting with Data Prepper version 2.0, Data Prepper no longer supports the `otel_traces_raw_prepper` processor. The `otel_traces_raw` processor replaces the `otel_traces_raw_prepper` processor and supports some of Data Prepper's recent data model changes. Instead, you should use the `otel_traces_raw` processor. See the following YAML file example:

```yml
entry-pipeline:
delay: "100"
source:
otel_trace_source:
otel_traces_source:
ssl: false
buffer:
bounded_blocking:
Expand All @@ -140,7 +140,7 @@
buffer_size: 10240
batch_size: 160
processor:
- otel_trace_raw:
- otel_traces_raw:
sink:
- opensearch:
hosts: ["https://localhost:9200"]
Expand Down Expand Up @@ -176,7 +176,7 @@

```yaml
source:
otel_trace_source:
otel_traces_source:
#record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0
ssl: true
sslKeyCertChainFile: "/full/path/to/certfile.crt"
Expand All @@ -202,7 +202,7 @@
# default value is 3_000 ms
delay: "100"
source:
otel_trace_source:
otel_traces_source:
#record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0
ssl: false # Change this to enable encryption in transit
authentication:
Expand Down Expand Up @@ -242,8 +242,8 @@
# With 64 as batch size each worker thread could process upto 3200 spans (64 * 50)
batch_size: 64
processor:
- otel_trace_raw:
- otel_trace_group:
- otel_traces_raw:
- otel_traces_group:
hosts: [ "https://localhost:9200" ]
# Change to your credentials
username: "admin"
Expand Down Expand Up @@ -369,9 +369,9 @@

Starting with Data Prepper version 1.4, trace processing uses Data Prepper's event model. This allows pipeline authors to configure other processors to modify spans or traces. To provide a migration path, Data Prepper version 1.4 introduced the following changes:

* `otel_trace_source` has an optional `record_type` parameter that can be set to `event`. When configured, it will output event objects.
* `otel_trace_raw` replaces `otel_trace_raw_prepper` for event-based spans.
* `otel_trace_group` replaces `otel_trace_group_prepper` for event-based spans.
* `otel_traces_source` has an optional `record_type` parameter that can be set to `event`. When configured, it will output event objects.
* `otel_traces_raw` replaces `otel_traces_raw_prepper` for event-based spans.
* `otel_traces_group` replaces `otel_traces_group_prepper` for event-based spans.

In Data Prepper version 2.0, `otel_trace_source` will only output events. Data Prepper version 2.0 also removes `otel_trace_raw_prepper` and `otel_trace_group_prepper` entirely. To migrate to Data Prepper version 2.0, you can configure your trace pipeline using the event model.
In Data Prepper version 2.0, `otel_traces_source` will only output events. Data Prepper version 2.0 also removes `otel_traces_raw_prepper` and `otel_traces_group_prepper` entirely. To migrate to Data Prepper version 2.0, you can configure your trace pipeline using the event model.

2 changes: 1 addition & 1 deletion _data-prepper/managing-data-prepper/core-apis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ authentication:

### Peer Forwarder

Peer Forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information about configuring Peer Forwarder, see [Peer forwarder]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/peer-forwarder/). It is supported by the `service_map_stateful`, `otel_trace_raw`, and `aggregate` processors.
Peer Forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information about configuring Peer Forwarder, see [Peer forwarder]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/peer-forwarder/). It is supported by the `service_map_stateful`, `otel_traces_raw`, and `aggregate` processors.

### Shutdown timeouts

Expand Down
4 changes: 2 additions & 2 deletions _data-prepper/managing-data-prepper/peer-forwarder.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@

# Peer forwarder
samipak458 marked this conversation as resolved.
Show resolved Hide resolved

Peer forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, peer forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).
Peer forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, peer forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_traces_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).

Check failure on line 10 in _data-prepper/managing-data-prepper/peer-forwarder.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'Peer Forwarder' instead of 'Peer forwarder'.

Raw Output:
{"message": "[Vale.Terms] Use 'Peer Forwarder' instead of 'Peer forwarder'.", "location": {"path": "_data-prepper/managing-data-prepper/peer-forwarder.md", "range": {"start": {"line": 10, "column": 1}}}, "severity": "ERROR"}

Check failure on line 10 in _data-prepper/managing-data-prepper/peer-forwarder.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'Peer Forwarder' instead of 'peer forwarder'.

Raw Output:
{"message": "[Vale.Terms] Use 'Peer Forwarder' instead of 'peer forwarder'.", "location": {"path": "_data-prepper/managing-data-prepper/peer-forwarder.md", "range": {"start": {"line": 10, "column": 304}}}, "severity": "ERROR"}
samipak458 marked this conversation as resolved.
Show resolved Hide resolved

Peer forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for peer forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys.
Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_traces_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys.

Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or AWS Cloud Map.

Expand Down
Loading