Adds ZSTD and ZSTD_NO_DICT codecs to core (out of sandbox)

_api-reference/index-apis/create-index.md

@@ -82,7 +82,14 @@ Setting | Description
 index.number_of_shards | The number of primary shards in the index. Default is 1.
 index.number_of_routing_shards | The number of routing shards used to split an index.
 index.shard.check_on_startup | Whether the index's shards should be checked for corruption. Available options are `false` (do not check for corruption), `checksum` (check for physical corruption), and `true` (check for both physical and logical corruption). Default is `false`.
-index.codec | The compression type to use to compress stored data. Available values are `default` (optimizes for retrieval speed) and `best_compression` (optimizes for better compression at the expense of speed, leading to smaller data sizes on disk). For snapshot distributions built with the sandbox feature enabled, `-Dsandbox.enabled=true`, OpenSearch offers a custom-codecs plugin that supports the value `zstd` for Zstandard compression.
+index.codec |  Determines how the index’s stored fields are compressed and stored on the disk. This setting impacts the size of the index shards and the performance of the operations on the index. Available values are:
+* `default` - This codec employs the 'LZ4' algorithm with a preset dictionary, which prioritizes performance over compression ratio. It offers faster indexing and search operations when compared with 'best_compression', but may result in larger index/shard sizes. If no codec is provided in the index settings, then 'LZ4'  is used as default algorithm for compression.
+* `best_compression` -  This codec utilizes 'zlib' as an underlying algorithm for compression. It achieves high compression ratios resulting in smaller index sizes. However, this may incur additional CPU usage during operations on the index and subsequently may result in high indexing and search latencies. 
+* `zstd` - This codec uses the ['Zstandard' compression algorithm](https://github.com/facebook/zstd), which provides a good balance between compression ratio and speed. It provides significant compression comparable to 'best_compression' codec with reasonable CPU usage and improved indexing/search performance comparable to 'default' codec.
+* `zstd_no_dict` This codec is similar to 'zstd' but excludes the dictionary compression feature. It provides faster indexing and search operations compared to 'zstd' at the expense of a slightly larger index size.
+
+'zstd' and 'zstd_no_dict' are the new codecs introduced in OpenSearch v2.9.0. They provide an option to configure the compression level as an index setting 'index.codec.compression_level' which is not available for other codecs. 
+{: .note}
 index.routing_partition_size | The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than just a single shard. To enable, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1.
 index.soft_deletes.retention_lease.period | The maximum amount of time to retain a shard's history of operations. Default is `12h`.
 index.load_fixed_bitset_filters_eagerly | Whether OpenSearch should pre-load cached filters. Available options are `true` and `false`. Default is `true`.

_install-and-configure/plugins.md

@@ -289,21 +289,6 @@ Members of the OpenSearch community have built countless plugins for the service
 | store-smb | 1.0.0 |
 | transport-nio | 1.0.0 |
 
-### Experimental plugins
-
-OpenSearch offers experimental plugins that may be used in a snapshot distribution that has the [sandbox feature enabled](https://github.com/opensearch-project/OpenSearch/blob/main/sandbox/build.gradle).
-
-| Plugin Name | Description | Earliest Available Version |
-| :--- | :--- |
-| custom-codecs | Provides additional compression codecs. | 1.0.0 |
-
-Use the following example command to enable the sandbox feature:
-
-```bash
-./gradlew assemble -Dsandbox.enabled=true
-bin/opensearch-plugin install file:///path/to/plugin-<version>-SNAPSHOT.zip
-```
-
 ## Related links
 
 - [About Observability]({{site.url}}{{site.baseurl}}/observability-plugin/index/)