-
Notifications
You must be signed in to change notification settings - Fork 504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds data prepper decompress processor documentation #6683
Changes from 2 commits
6ef8f09
422c3a6
d0276a8
e020723
00d8127
b2aa2d6
2f01757
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
--- | ||
layout: default | ||
title: decompress | ||
parent: Processors | ||
grand_parent: Pipelines | ||
nav_order: 40 | ||
--- | ||
|
||
# decompress | ||
Check failure on line 9 in _data-prepper/pipelines/configuration/processors/decompress.md GitHub Actions / style-job
|
||
|
||
The `decompress` processor decompresses the `base64`-encoded compressed fields inside of an event. | ||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Configuration | ||
|
||
Option | Required | Type | Description | ||
:--- | :--- | :--- | :--- | ||
`keys` | Yes | List<String> | The fields in the event that will be decompressed. | ||
`type` | Yes | Enum | The type of decompression to use for the `keys` specified. As of Data Prepper 2.7, only `gzip` is supported. `decompress_when` | No | String| A [Data Prepper conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/) to determine which events to run the `decompress` processor. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "As of Data Prepper 2.7" - Do we need this? That is the latest version. I'd just say "Only |
||
`tags_on_failure` | No | List<String> | A list of strings to tag events with when the processor fails to decompress the `keys` on the event. Defaults to `_decompression_failure`. | ||
|
||
## Usage | ||
|
||
The following example shows the `decompress` processor when used in `pipelines.yaml`: | ||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```yaml | ||
processor: | ||
- decompress: | ||
decompress_when: '/some_key == null' | ||
keys: [ "base_64_gzip_key" ] | ||
type: gzip | ||
``` | ||
|
||
## Metrics | ||
|
||
The following table describes common [Abstract processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/processor/AbstractProcessor.java) metrics. | ||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
| Metric name | Type | Description | | ||
| ------------- | ---- | -----------| | ||
| `recordsIn` | Counter | The ingress of records to a pipeline component. | | ||
| `recordsOut` | Counter | The egress of records from a pipeline component. | | ||
| `timeElapsed` | Timer | The time elapsed during execution of a pipeline component. | | ||
|
||
The `decompress` processor includes the following custom metrics. | ||
|
||
### Counter | ||
|
||
* `processingErrors`: The number of processing errors that have occurred in the `decompress` processor. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably don't need backticks around "base64" as it is not a keyword in this context.