Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) #4421

sarthakaggarwal97 · 2023-06-29T10:26:48Z

Description

ZSTD and ZSTD_NO_DICT codecs were added to core and out of the sandbox as a part of this pull request
We would need to change the documentation once this is closed.
The codecs will be available without the sandbox plugin as a part of OS 2.9 release.

Issues Resolved

#3816 (partly)

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sarthak Aggarwal <[email protected]>

hdhalter · 2023-06-29T14:39:03Z

@Naarcha-AWS - since this issue is assigned to you, can you please review the PR? Thanks!

_api-reference/index-apis/create-index.md

Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Heather Halter <[email protected]>

_api-reference/index-apis/create-index.md

Signed-off-by: Heather Halter <[email protected]>

hdhalter · 2023-07-12T15:02:23Z

@sarthakaggarwal97 - can you please review? Is there anything else I can add to differentiate the options? Thanks.

sarthakaggarwal97 · 2023-07-12T15:26:01Z

@sarthakaggarwal97 - can you please review? Is there anything else I can add to differentiate the options? Thanks.

@hdhalter @Naarcha-AWS we can add the information related regarding the different codecs using different algorithms. default -> lz4 algorithm, best_compression -> uses zlib algorithm, and zstd/zstd_no_dict uses Zstandard algorithm.

Also, there is a separate documentation is being prepared to help users understand more about index.codec setting, which includes that zstd_no_dict provides the best performance with indexing operations.

hdhalter

@sarthakaggarwal97 - please review the proposed changes. Should I also move that section down to the dynamic table?

_api-reference/index-apis/create-index.md

Signed-off-by: Heather Halter <[email protected]>

_api-reference/index-apis/create-index.md

Signed-off-by: Heather Halter <[email protected]>

hdhalter

Fixed a tick mark

_api-reference/index-apis/create-index.md

Signed-off-by: Heather Halter <[email protected]>

Fixed the code samples Signed-off-by: Heather Halter <[email protected]>

hdhalter

Fixed broken link

_api-reference/index-apis/create-index.md

Signed-off-by: Heather Halter <[email protected]>

Added back a heading 'Index Settings' since it caused broken links in other topics. Signed-off-by: Heather Halter <[email protected]>

sarthakaggarwal97 · 2023-07-14T05:19:06Z

_api-reference/index-apis/create-index.md

@@ -94,6 +91,7 @@ Setting | Description
 :--- | :---
 index.number_of_replicas | The number of replica shards each primary shard should have. For example, if you have 4 primary shards and set `index.number_of_replicas` to 3, the index has 12 replica shards. Default is 1.
 index.auto_expand_replicas | Whether the cluster should automatically add replica shards based on the number of data nodes. Specify a lower bound and upper limit (for example, 0-9), or `all` for the upper limit. For example, if you have 5 data nodes and set `index.auto_expand_replicas` to 0-3, then the cluster does not automatically add another replica shard. However, if you set this value to `0-all` and add 2 more nodes for a total of 7, the cluster will expand to now have 6 replica shards. Default is disabled.
+index.codec |  Determines how the index’s stored fields are compressed and stored on the disk. This setting impacts the size of the index shards and the performance of the operations on the index. Available values are: `default', 'best_compression`, `zstd` and `zstd_no_dict`. Two new codecs are introduced in OpenSearch 2.9: `zstd` and `zstd_no_dict`. They provide an option to configure the compression level as an index setting `index.codec.compression_level` which is not available for other codecs. For information about each setting, see [Index codec settings](#Index-codec-settings).


Can we add a line over here that index.codec settings cannot be updated in real-time.
Similarly, index.codec.compression_level can also not be updated at real time.

sarthakaggarwal97 · 2023-07-14T05:22:25Z

_api-reference/index-apis/create-index.md

+index.final_pipeline | The final ingest node pipeline for the index. If the final pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
+
+#### Index codec settings 


As discussed offline, we will add an additional page to explain the index.codec settings around the time of release.
This information might be redundant post the addition of that page, how do you suggest we should proceed @hdhalter

Calling it out here: We might need to update the documentation related to snapshot, and we can check if we can add something over here at best practices for performance

We need to add more information about indexing, in general, and will add more information to support these options after the release.

Can you please provide me an endpoint so I can test the code samples on Monday? Thanks!

@sarthakaggarwal97 - The documentation related to snapshots has been updated. The link above to 'best practices' is part of the Service documentation and needs to be handled by @chatnish.

Can you please provide me an endpoint so I can test the code samples on Monday? Thanks!

Do you want a domain where you can test these code samples?

Added the compression_level setting to the dynamic table. Signed-off-by: Heather Halter <[email protected]>

Added sentences about not being able to be updated dynamically. Signed-off-by: Heather Halter <[email protected]>

natebower

@hdhalter Please see my changes and let me know if you have any questions. Thanks!

_api-reference/index-apis/create-index.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

sarthakaggarwal97 · 2023-07-18T13:15:18Z

LGTM!

hdhalter · 2023-07-19T18:12:43Z

Confirmed with @sarthakaggarwal97 this is good to go. @kolchfa-aws will take over the updates post-2.9.

…h-project#4421) * documentation for zstd and zstd out of sandbox Signed-off-by: Sarthak Aggarwal <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added a new section with codec information Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Fixed the code samples Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added back a heading 'Index Settings' since it caused broken links in other topics. Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added the compression_level setting to the dynamic table. Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added sentences about not being able to be updated dynamically. Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* documentation for zstd and zstd out of sandbox Signed-off-by: Sarthak Aggarwal <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added a new section with codec information Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Fixed the code samples Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added back a heading 'Index Settings' since it caused broken links in other topics. Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added the compression_level setting to the dynamic table. Signed-off-by: Heather Halter <[email protected]> * Update create-index.md Added sentences about not being able to be updated dynamically. Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Update _api-reference/index-apis/create-index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

documentation for zstd and zstd out of sandbox

90e4dcf

Signed-off-by: Sarthak Aggarwal <[email protected]>

sarthakaggarwal97 requested review from cwillum, hdhalter, kolchfa-aws, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners June 29, 2023 10:26

sarthakaggarwal97 changed the title ~~documentation for zstd and zstd out of sandbox~~ Documentation for zstd and zstd out of sandbox Jun 29, 2023

Naarcha-AWS requested changes Jul 3, 2023

View reviewed changes

_api-reference/index-apis/create-index.md Outdated Show resolved Hide resolved

Naarcha-AWS self-assigned this Jul 3, 2023

Naarcha-AWS added the v2.9.0 label Jul 6, 2023

hdhalter added the release-notes PR: Include this PR in the automated release notes label Jul 7, 2023

hdhalter changed the title ~~Documentation for zstd and zstd out of sandbox~~ Adds ZSTD and ZSTD_NO_DICT codecs out of sandbox to core Jul 11, 2023

hdhalter changed the title ~~Adds ZSTD and ZSTD_NO_DICT codecs out of sandbox to core~~ Adds ZSTD and ZSTD_NO_DICT codecs to core (out of standbox) Jul 11, 2023

Update _api-reference/index-apis/create-index.md

969d3fb

Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Heather Halter <[email protected]>

hdhalter approved these changes Jul 11, 2023

View reviewed changes

_api-reference/index-apis/create-index.md Outdated Show resolved Hide resolved

hdhalter changed the title ~~Adds ZSTD and ZSTD_NO_DICT codecs to core (out of standbox)~~ Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) Jul 11, 2023

Update _api-reference/index-apis/create-index.md

f8bc687

Signed-off-by: Heather Halter <[email protected]>

hdhalter added the 3 - Tech review PR: Tech review in progress label Jul 12, 2023

hdhalter reviewed Jul 13, 2023

View reviewed changes

_api-reference/index-apis/create-index.md Outdated Show resolved Hide resolved

Update _api-reference/index-apis/create-index.md

57c46b0

Signed-off-by: Heather Halter <[email protected]>

hdhalter reviewed Jul 13, 2023

View reviewed changes

hdhalter added 2 commits July 13, 2023 13:50

Update _api-reference/index-apis/create-index.md

6c8c087

Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

642f364

Signed-off-by: Heather Halter <[email protected]>

hdhalter reviewed Jul 13, 2023

View reviewed changes

_api-reference/index-apis/create-index.md Outdated Show resolved Hide resolved

hdhalter added 2 commits July 13, 2023 16:28

Update _api-reference/index-apis/create-index.md

793db12

Signed-off-by: Heather Halter <[email protected]>

Update create-index.md

a050110

Fixed the code samples Signed-off-by: Heather Halter <[email protected]>

hdhalter reviewed Jul 13, 2023

View reviewed changes

_api-reference/index-apis/create-index.md Outdated Show resolved Hide resolved

hdhalter added 2 commits July 13, 2023 16:44

Update _api-reference/index-apis/create-index.md

8ae0f2d

Signed-off-by: Heather Halter <[email protected]>

Update create-index.md

3a0e510

Added back a heading 'Index Settings' since it caused broken links in other topics. Signed-off-by: Heather Halter <[email protected]>

sarthakaggarwal97 commented Jul 14, 2023

View reviewed changes

hdhalter added 2 commits July 14, 2023 09:45

Update create-index.md

a66b865

Added the compression_level setting to the dynamic table. Signed-off-by: Heather Halter <[email protected]>

Update create-index.md

7046371

Added sentences about not being able to be updated dynamically. Signed-off-by: Heather Halter <[email protected]>

Naarcha-AWS approved these changes Jul 14, 2023

View reviewed changes

hdhalter added 5 - Editorial review PR: Editorial review in progress and removed 3 - Tech review PR: Tech review in progress labels Jul 14, 2023

natebower reviewed Jul 14, 2023

View reviewed changes

hdhalter and others added 8 commits July 14, 2023 13:32

Update _api-reference/index-apis/create-index.md

bfc1d32

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

ea7964f

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

9e4509a

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

e4e7846

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

988f4df

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

c2feee0

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

6b4fad4

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Update _api-reference/index-apis/create-index.md

3ade466

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

hdhalter self-assigned this Jul 17, 2023

hdhalter added 6 - Done but waiting to merge PR: The work is done and ready to merge and removed 5 - Editorial review PR: Editorial review in progress labels Jul 17, 2023

hdhalter merged commit de4276a into opensearch-project:main Jul 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) #4421

Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) #4421

sarthakaggarwal97 commented Jun 29, 2023 •

edited

Loading

hdhalter commented Jun 29, 2023

hdhalter commented Jul 12, 2023

sarthakaggarwal97 commented Jul 12, 2023 •

edited

Loading

hdhalter left a comment

hdhalter left a comment

hdhalter left a comment

sarthakaggarwal97 Jul 14, 2023 •

edited

Loading

sarthakaggarwal97 Jul 14, 2023

sarthakaggarwal97 Jul 14, 2023

hdhalter Jul 14, 2023

hdhalter Jul 14, 2023

hdhalter Jul 17, 2023

sarthakaggarwal97 Jul 18, 2023

natebower left a comment

sarthakaggarwal97 commented Jul 18, 2023

hdhalter commented Jul 19, 2023

		index.final_pipeline \| The final ingest node pipeline for the index. If the final pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.

		#### Index codec settings

Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) #4421

Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox) #4421

Conversation

sarthakaggarwal97 commented Jun 29, 2023 • edited Loading

Description

Issues Resolved

Checklist

hdhalter commented Jun 29, 2023

hdhalter commented Jul 12, 2023

sarthakaggarwal97 commented Jul 12, 2023 • edited Loading

hdhalter left a comment

Choose a reason for hiding this comment

hdhalter left a comment

Choose a reason for hiding this comment

hdhalter left a comment

Choose a reason for hiding this comment

sarthakaggarwal97 Jul 14, 2023 • edited Loading

Choose a reason for hiding this comment

sarthakaggarwal97 Jul 14, 2023

Choose a reason for hiding this comment

sarthakaggarwal97 Jul 14, 2023

Choose a reason for hiding this comment

hdhalter Jul 14, 2023

Choose a reason for hiding this comment

hdhalter Jul 14, 2023

Choose a reason for hiding this comment

hdhalter Jul 17, 2023

Choose a reason for hiding this comment

sarthakaggarwal97 Jul 18, 2023

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

sarthakaggarwal97 commented Jul 18, 2023

hdhalter commented Jul 19, 2023

sarthakaggarwal97 commented Jun 29, 2023 •

edited

Loading

sarthakaggarwal97 commented Jul 12, 2023 •

edited

Loading

sarthakaggarwal97 Jul 14, 2023 •

edited

Loading