From 78247f84fbd84ad67c1ca39c1fa7cb2525d9fe37 Mon Sep 17 00:00:00 2001 From: Hai Yan <8153134+oeyh@users.noreply.github.com> Date: Thu, 12 Oct 2023 08:53:20 -0500 Subject: [PATCH] Add documentation for Data Prepper translate processor (#5122) * Add translate processor docs Signed-off-by: Hai Yan * Fix a table display issue Signed-off-by: Hai Yan * Address review comments: tweak sentences; add type info; add top level options Signed-off-by: Hai Yan * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/processors/translate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update translate.md * Update translate.md * Update _data-prepper/pipelines/configuration/processors/translate.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Hai Yan Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- .../configuration/processors/otel-metrics.md | 2 +- .../configuration/processors/translate.md | 176 ++++++++++++++++++ 2 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 _data-prepper/pipelines/configuration/processors/translate.md diff --git a/_data-prepper/pipelines/configuration/processors/otel-metrics.md b/_data-prepper/pipelines/configuration/processors/otel-metrics.md index dacabd2f14..08fb72810e 100644 --- a/_data-prepper/pipelines/configuration/processors/otel-metrics.md +++ b/_data-prepper/pipelines/configuration/processors/otel-metrics.md @@ -8,7 +8,7 @@ nav_order: 72 # otel_metrics -The `otel_metrics` processor serializes a collection of `ExportMetricsServiceRequest` records sent from the [OTel metrics source]({{site.url}}{{site.baseurl}}//data-prepper/pipelines/configuration/sources/otel-metrics-source/) into a collection of string records. +The `otel_metrics` processor serializes a collection of `ExportMetricsServiceRequest` records sent from the [OTel metrics source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-metrics-source/) into a collection of string records. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/translate.md b/_data-prepper/pipelines/configuration/processors/translate.md new file mode 100644 index 0000000000..d29aa5894c --- /dev/null +++ b/_data-prepper/pipelines/configuration/processors/translate.md @@ -0,0 +1,176 @@ +--- +layout: default +title: translate +parent: Processors +grand_parent: Pipelines +nav_order: 117 +--- + +# translate + +The `translate` processor transforms values in events into preconfigured values. + +## Basic usage + +To use the `translate` processor, create the following `pipeline.yaml` file: + +```yaml +translate-pipeline: + source: + file: + path: "/full/path/to/logs_json.log" + record_type: "event" + format: "json" + processor: + - translate: + mappings: + - source: "status" + targets: + - target: "translated_result" + map: + 404: "Not Found" + sink: + - stdout: +``` + +Then create the following file named `logs_json.log` and replace the `path` in the file source of your `pipeline.yaml` file with the path of a file containing the following JSON data: + +```json +{ "status": "404" } +``` + +The `translate` processor configuration in `pipeline.yaml` retrieves the `source` value from the event data and compares it against the keys specified under the `targets`. +When a match is found, the processor places the corresponding mapped value into the `target` key provided in the configuration. + +When you run Data Prepper with the previous `pipeline.yaml` file, you should receive the following output: + +```json +{ + "status": "404", + "translated_result": "Not Found" +} +``` + +## Advanced options + +The following example shows a more involved mapping with additional configurations for the `translate` processor: + +```yaml +processor: + - translate: + mappings: + - source: "status" + targets: + - target: "translated_result" + map: + 404: "Not Found" + default: "default" + type: "string" + translate_when: "/response != null" + - target: "another_translated_result" + regex: + exact: false + patterns: + "2[0-9]{2}" : "Success" # Matches ranges from 200-299 + "5[0-9]{2}": "Error" # Matches ranges form 500-599 + file: + name: "path/to/file.yaml" + aws: + bucket: my_bucket + region: us-east-1 + sts_role_arn: arn:aws:iam::123456789012:role/MyS3Role +``` + +On the top level, specify `mappings` for inline mapping configurations, or `file` pull mapping configurations from a file. Both `mappings` and `file` options can be specified together, and the processor considers the mappings from both sources for translations. In instances where the pipeline configuration and file mappings share duplicate `source` and `target` pairs, the mappings specified within the pipeline configuration take precedence. + + +## Configuration + +You can use the following options to configure the `translate` processor. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| mappings | No | List | Defines inline mappings. For more information, see [mappings](#mappings). | +| file | No | Map | Points to the file that contains mapping configurations. For more information, see [file](#file). | + +### mappings + +Each item in the `mappings` configuration contains the following options. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| source | Yes | String or list | The source field to translate. Can be a string or a list of strings. | +| targets | Yes | List | A list of target field configurations, such as the target field key or translation maps. | + +Each item in the `targets` configuration contains the following options. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| target | Yes | String | The key that specifies the field in the output in which the translated value will be placed. | +| map | No | Map | A list of key-value pairs that define the translations. Each key represents a possible value in the source field, and the corresponding value represents what it should be translated to. For examples, see [map option](#map-option). At least one of `map` and `regex` should be configured. | +| regex | No | Map | A map of keys that defines the translation map. For more options, see [regex option](#regex-option). At least one of `map` and `regex` should be configured. | +| default | No | String | The default value to use when no match is found during translation. | +| type | No | String | Specifies the data type for the target value. | +| translate_when | No | String | Uses a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) to specify a condition for performing the translation. When specified, the expression will only translate when the condition is met. | + +#### map option + +You can use the following key types when using the map option: + +* Individual keys + ```yaml + map: + ok : "Success" + 120: "Found" + ``` +* Number ranges + ```yaml + map: + "100-200": "Success" + "400-499": "Error" + ``` +* Comma-delimited keys + ```yaml + map: + "key1,key2,key3": "value1" + "100-200,key4": "value2" + ``` + +When configuring the keys inside the `map` option, do not use any overlapping number ranges or duplicate keys. + +#### regex option + +You can use the following options with the `regex` option. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| patterns | Yes | Map | A map of key-value pairs defining the regex patterns of keys and the value to translate to for each pattern. | +| exact | No | Boolean | Whether to use full string match or partial string match on the regex pattern. If `true`, the pattern is considered a match only when the entire key matches the pattern. Otherwise, the pattern is considered a match when a sub-string of the key matches the pattern. | + +### file + +The `file` option in the `translate` processor takes a local YAML file or an Amazon Simple Storage Service (Amazon S3) object containing translation mappings. The file's contents should be in the following format: +```yaml +mappings: + - source: "status" + targets: + - target: "result" + map: + "foo": "bar" + # Other configurations +``` + +You can use the following options in the `file` configuration. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| name | Yes | String | The full path to a local file or key name for an S3 object. | +| aws | No | Map | The AWS configuration when the file is an S3 object. See the following table for more information. | + +You can use the following options with the `aws` configuration. + +| Parameter | Required | Type | Description | +| :--- | :--- | :--- | :--- | +| `bucket` | Yes | String | The Amazon S3 bucket name. | +| `region` | Yes | String | The AWS Region to use for credentials. | +| `sts_role_arn` | Yes | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3. |