Adds data prepper decompress processor documentation #6683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Taylor Gray <[email protected]>
graytaylor0
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
AMoo-Miki,
natebower and
dlvenable
as code owners
Signed-off-by: Naarcha-AWS <[email protected]>
Naarcha-AWS
added
3 - Tech review
Naarcha-AWS
approved these changes
Mar 18, 2024
dlvenable
reviewed
Mar 19, 2024
There was a problem hiding this comment.
@graytaylor0 , @Naarcha-AWS , I have a couple of small comments. But, other than that I'm good to proceed.
|
|
||
| # decompress | ||
|
|
||
| The `decompress` processor decompresses the `base64`-encoded compressed fields inside of an event. |
There was a problem hiding this comment.
We probably don't need backticks around "base64" as it is not a keyword in this context.
| Option | Required | Type | Description | ||
| :--- | :--- | :--- | :--- | ||
| `keys` | Yes | List<String> | The fields in the event that will be decompressed. | ||
| `type` | Yes | Enum | The type of decompression to use for the `keys` specified. As of Data Prepper 2.7, only `gzip` is supported. `decompress_when` | No | String| A [Data Prepper conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/) to determine which events to run the `decompress` processor. |
There was a problem hiding this comment.
"As of Data Prepper 2.7" - Do we need this? That is the latest version. I'd just say "Only gzip is supported" and we can expand later.
Naarcha-AWS
reviewed
Mar 19, 2024
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Naarcha-AWS
added
5 - Editorial review
natebower
reviewed
Mar 19, 2024
There was a problem hiding this comment.
@graytaylor0 @Naarcha-AWS Just a few minor changes. Thanks!
Naarcha-AWS
reviewed
Mar 20, 2024
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>
Naarcha-AWS
merged commit 072d506
into
opensearch-project:data-prepper-2.7
2 of 3 checks passed
hdhalter
added
3 - Done
Naarcha-AWS
added a commit
that referenced
this pull request
Mar 27, 2024
* Obfuscate processor doc (#6387) Signed-off-by: shaavanga <[email protected]> * [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384) * MAINT: document on disable_refresh secret extension setting Signed-off-by: George Chen <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Updates to the S3 source documentation (#6379) * Updates to the S3 source documentation to include missing features and metrics. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Try to fix build error * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * See if removing links removes build error * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update list_to_map processor in Data Prepper (#6382) * Update list-to-map processor Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Improvements to the S3 sink documenation (#6383) * Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema. Signed-off-by: David Venable <[email protected]> * Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist. Signed-off-by: David Venable <[email protected]> * Update s3.md * Update s3.md * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/s3.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * Add David's link Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380) * Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation Signed-off-by: Taylor Gray <[email protected]> * Address PR comments Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update dynamo-db.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Date Processor doc update (#6381) * Date Processor doc update Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fixed table header indentation Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fix formatting and grammar. * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update date.md --------- Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for join function in Data Prepper (#6688) * Add docs for join function Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update expression-syntax.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for flatten processor for Data Prepper (#6685) * Add docs for flatten processor Signed-off-by: Hai Yan <[email protected]> * Add grammar edits * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update flatten.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds data prepper decompress processor documentation (#6683) * Add data prepper decompress processor documentation Signed-off-by: Taylor Gray <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add data prepper documentation for grok performance_metadata (#6681) * Add data prepper grok performance metadata documentation Signed-off-by: Taylor Gray <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Documentation for data prepper dynamodb source view_on_remove feature (#6738) * Add documentation for dynamodb source view_on_remove feature Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Update Data Prepper opensearch sink documentation (#6386) * Update Data Prepper opensearch sink documentation Signed-off-by: Taylor Gray <[email protected]> * Formatting fixes and adding introductory text. Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds a configuration for the Data Prepper S3 source workers field. (#6774) * Adds a configuration for the Data Prepper S3 source workers field. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Add parse_ion processor (#6761) * Add parse_ion processor Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Co-authored-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update parse-ion.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for map_to_list processor (#6680) * Add docs for map_to_list processor Signed-off-by: Hai Yan <[email protected]> * Update map-to-list.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772) * Adds documentation for the Data Prepper geoip processor and geoip_service extension. Signed-off-by: David Venable <[email protected]> * Update extensions.md * Update geoip_service.md * Update geoip.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/extensions/extensions.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update flatten.md Signed-off-by: Naarcha-AWS <[email protected]> * Fix nav order Signed-off-by: Archer <[email protected]> --------- Signed-off-by: shaavanga <[email protected]> Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Archer <[email protected]> Co-authored-by: Prathyusha Vangala <[email protected]> Co-authored-by: Qi Chen <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Asif Sohail Mohammed <[email protected]> Co-authored-by: Heather Halter <[email protected]>
Naarcha-AWS
added a commit
that referenced
this pull request
Mar 27, 2024
* Obfuscate processor doc (#6387) Signed-off-by: shaavanga <[email protected]> * [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384) * MAINT: document on disable_refresh secret extension setting Signed-off-by: George Chen <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Updates to the S3 source documentation (#6379) * Updates to the S3 source documentation to include missing features and metrics. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Try to fix build error * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * See if removing links removes build error * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update list_to_map processor in Data Prepper (#6382) * Update list-to-map processor Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Improvements to the S3 sink documenation (#6383) * Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema. Signed-off-by: David Venable <[email protected]> * Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist. Signed-off-by: David Venable <[email protected]> * Update s3.md * Update s3.md * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/s3.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * Add David's link Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380) * Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation Signed-off-by: Taylor Gray <[email protected]> * Address PR comments Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update dynamo-db.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Date Processor doc update (#6381) * Date Processor doc update Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fixed table header indentation Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fix formatting and grammar. * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update date.md --------- Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for join function in Data Prepper (#6688) * Add docs for join function Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update expression-syntax.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for flatten processor for Data Prepper (#6685) * Add docs for flatten processor Signed-off-by: Hai Yan <[email protected]> * Add grammar edits * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update flatten.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds data prepper decompress processor documentation (#6683) * Add data prepper decompress processor documentation Signed-off-by: Taylor Gray <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add data prepper documentation for grok performance_metadata (#6681) * Add data prepper grok performance metadata documentation Signed-off-by: Taylor Gray <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Documentation for data prepper dynamodb source view_on_remove feature (#6738) * Add documentation for dynamodb source view_on_remove feature Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Update Data Prepper opensearch sink documentation (#6386) * Update Data Prepper opensearch sink documentation Signed-off-by: Taylor Gray <[email protected]> * Formatting fixes and adding introductory text. Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds a configuration for the Data Prepper S3 source workers field. (#6774) * Adds a configuration for the Data Prepper S3 source workers field. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Add parse_ion processor (#6761) * Add parse_ion processor Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Co-authored-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update parse-ion.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for map_to_list processor (#6680) * Add docs for map_to_list processor Signed-off-by: Hai Yan <[email protected]> * Update map-to-list.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772) * Adds documentation for the Data Prepper geoip processor and geoip_service extension. Signed-off-by: David Venable <[email protected]> * Update extensions.md * Update geoip_service.md * Update geoip.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/extensions/extensions.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update flatten.md Signed-off-by: Naarcha-AWS <[email protected]> * Fix nav order Signed-off-by: Archer <[email protected]> --------- Signed-off-by: shaavanga <[email protected]> Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Archer <[email protected]> Co-authored-by: Prathyusha Vangala <[email protected]> Co-authored-by: Qi Chen <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Asif Sohail Mohammed <[email protected]> Co-authored-by: Heather Halter <[email protected]> (cherry picked from commit 48651b0)
1 task
Naarcha-AWS
added a commit
that referenced
this pull request
Mar 27, 2024
* Obfuscate processor doc (#6387) Signed-off-by: shaavanga <[email protected]> * [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384) * MAINT: document on disable_refresh secret extension setting Signed-off-by: George Chen <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/configuring-data-prepper.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Updates to the S3 source documentation (#6379) * Updates to the S3 source documentation to include missing features and metrics. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Try to fix build error * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Update s3.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * See if removing links removes build error * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update list_to_map processor in Data Prepper (#6382) * Update list-to-map processor Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Improvements to the S3 sink documenation (#6383) * Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema. Signed-off-by: David Venable <[email protected]> * Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist. Signed-off-by: David Venable <[email protected]> * Update s3.md * Update s3.md * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/sinks/s3.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update s3.md * Add David's link Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380) * Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation Signed-off-by: Taylor Gray <[email protected]> * Address PR comments Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update dynamo-db.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Date Processor doc update (#6381) * Date Processor doc update Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fixed table header indentation Signed-off-by: Asif Sohail Mohammed <[email protected]> * Fix formatting and grammar. * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update date.md --------- Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for join function in Data Prepper (#6688) * Add docs for join function Signed-off-by: Hai Yan <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update expression-syntax.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for flatten processor for Data Prepper (#6685) * Add docs for flatten processor Signed-off-by: Hai Yan <[email protected]> * Add grammar edits * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update flatten.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds data prepper decompress processor documentation (#6683) * Add data prepper decompress processor documentation Signed-off-by: Taylor Gray <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Update decompress.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add data prepper documentation for grok performance_metadata (#6681) * Add data prepper grok performance metadata documentation Signed-off-by: Taylor Gray <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update grok.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Documentation for data prepper dynamodb source view_on_remove feature (#6738) * Add documentation for dynamodb source view_on_remove feature Signed-off-by: Taylor Gray <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Update Data Prepper opensearch sink documentation (#6386) * Update Data Prepper opensearch sink documentation Signed-off-by: Taylor Gray <[email protected]> * Formatting fixes and adding introductory text. Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Update opensearch.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds a configuration for the Data Prepper S3 source workers field. (#6774) * Adds a configuration for the Data Prepper S3 source workers field. Signed-off-by: David Venable <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> * Add parse_ion processor (#6761) * Add parse_ion processor Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Co-authored-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update parse-ion.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Add docs for map_to_list processor (#6680) * Add docs for map_to_list processor Signed-off-by: Hai Yan <[email protected]> * Update map-to-list.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772) * Adds documentation for the Data Prepper geoip processor and geoip_service extension. Signed-off-by: David Venable <[email protected]> * Update extensions.md * Update geoip_service.md * Update geoip.md * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/managing-data-prepper/extensions/extensions.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: David Venable <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> * Update flatten.md Signed-off-by: Naarcha-AWS <[email protected]> * Fix nav order Signed-off-by: Archer <[email protected]> --------- Signed-off-by: shaavanga <[email protected]> Signed-off-by: George Chen <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Hai Yan <[email protected]> Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Asif Sohail Mohammed <[email protected]> Signed-off-by: Archer <[email protected]> Co-authored-by: Prathyusha Vangala <[email protected]> Co-authored-by: Qi Chen <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Asif Sohail Mohammed <[email protected]> Co-authored-by: Heather Halter <[email protected]> (cherry picked from commit 48651b0)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds documentations for data prepper's new
decompressprocessor.Issues Resolved
Closes #6407
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.