Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add split event processor for dataprepper #6682

Merged
merged 6 commits into from
Mar 27, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions _data-prepper/pipelines/configuration/processors/split-event.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
layout: default
title: split-event
parent: Processors
grand_parent: Pipelines
nav_order: 56
---

# split-event

Check failure on line 9 in _data-prepper/pipelines/configuration/processors/split-event.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'split-event' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'split-event' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/configuration/processors/split-event.md", "range": {"start": {"line": 9, "column": 3}}}, "severity": "ERROR"}

The `split-event` processor is used to split events based on a delimiter and generates multiple events from a user-specified field.

## Configuration

The following table describes the configuration options for the `split-event` processor.

| Option | Type | Description |
|------------------|---------|-----------------------------------------------------------------------------------------------|
| `field` | String | The event field to be split. |
| `delimiter_regex`| String | The regular expression used as the delimiter for splitting the field. |
| `delimiter` | String | The delimiter used for splitting the field. If not specified, the default delimiter is used. |

# Usage

To use the `split-event` processor, add the following to your `pipelines.yaml` file:

```
split-event-pipeline:
source:
http:
processor:
- split_event:
field: query
delimiter: ' '
sink:
- stdout:
```
{% include copy.html %}

When an event contains the following example input:

```
{"query" : "open source", "some_other_field" : "abc" }
```

The input will be split into multiple events based on the `query` field, with the delimiter set as white space, as shown in the following example:

```
{"query" : "open", "some_other_field" : "abc" }
{"query" : "source", "some_other_field" : "abc" }
```

Loading