Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds data prepper decompress processor documentation #6683

Merged
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions _data-prepper/pipelines/configuration/processors/decompress.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
layout: default
title: decompress
parent: Processors
grand_parent: Pipelines
nav_order: 40
---

# decompress

Check failure on line 9 in _data-prepper/pipelines/configuration/processors/decompress.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'decompress' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'decompress' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/configuration/processors/decompress.md", "range": {"start": {"line": 9, "column": 3}}}, "severity": "ERROR"}

The `decompress` processor decompresses the `base64`-encoded compressed fields inside of an event.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't need backticks around "base64" as it is not a keyword in this context.

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Configuration

Option | Required | Type | Description
:--- | :--- | :--- | :---
`keys` | Yes | List<String> | The fields in the event that will be decompressed.
`type` | Yes | Enum | The type of decompression to use for the `keys` specified. As of Data Prepper 2.7, only `gzip` is supported. `decompress_when` | No | String| A [Data Prepper conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/) to determine which events to run the `decompress` processor.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"As of Data Prepper 2.7" - Do we need this? That is the latest version. I'd just say "Only gzip is supported" and we can expand later.

`tags_on_failure` | No | List<String> | A list of strings to tag events with when the processor fails to decompress the `keys` on the event. Defaults to `_decompression_failure`.

## Usage

The following example shows the `decompress` processor when used in `pipelines.yaml`:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```yaml
processor:
- decompress:
decompress_when: '/some_key == null'
keys: [ "base_64_gzip_key" ]
type: gzip
```

## Metrics

The following table describes common [Abstract processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/processor/AbstractProcessor.java) metrics.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

| Metric name | Type | Description |
| ------------- | ---- | -----------|
| `recordsIn` | Counter | The ingress of records to a pipeline component. |
| `recordsOut` | Counter | The egress of records from a pipeline component. |
| `timeElapsed` | Timer | The time elapsed during execution of a pipeline component. |

The `decompress` processor includes the following custom metrics.

### Counter

* `processingErrors`: The number of processing errors that have occurred in the `decompress` processor.

Loading