Introducing ZStd compression codec plugin #9658

sarthakaggarwal97 · 2023-08-31T08:50:53Z

Description

Introduces ZStd Compression Codecs as a plugin

Related Issues

Resolves #9422

Tests Covered:

Long running tests to verify for any corruption or memory leaks
Backward compatibility test to ensure that the segments that are created in OS 2.9, are readable in OS 2.10 with ZStd as a plugin.
Reindexing between different compression codecs
Create a snapshot, delete an index, restore from snapshot
Split/Shrink api since they modify segments to some extent
Index Blocks
Upgrade from the minor version
Force merge of segments with multiple compression mode into zstd
Close an index, update the setting, test the indexing and search flows, snapshots, etc i.e. with multiple compression

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-31T08:55:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23943/
CommitID: 4218b0e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T09:16:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23946/
CommitID: 3d2835b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T09:25:36Z

Compatibility status:

Checks if related components are compatible with change 4218b0e

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T09:38:11Z

Compatibility status:

Checks if related components are compatible with change 3d2835b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T18:18:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23992/
CommitID: e9dc084
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T18:45:23Z

Compatibility status:

Checks if related components are compatible with change e9dc084

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T18:46:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23995/
CommitID: 4ff3aac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T18:56:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23996/
CommitID: 0d0aea6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T19:07:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24001/
CommitID: 42fdd85
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T19:12:59Z

Compatibility status:

Checks if related components are compatible with change 0d0aea6

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T19:16:07Z

Compatibility status:

Checks if related components are compatible with change 4ff3aac

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T19:17:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24003/
CommitID: 8bd83a6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T19:30:38Z

Compatibility status:

Checks if related components are compatible with change 42fdd85

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T19:32:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24007/
CommitID: f54a80e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T19:34:24Z

Compatibility status:

Checks if related components are compatible with change 8bd83a6

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T19:57:10Z

Compatibility status:

Checks if related components are compatible with change f54a80e

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T20:03:20Z

Compatibility status:

Checks if related components are compatible with change a9512b5

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T20:08:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24008/
CommitID: a9512b5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-06T08:37:27Z

Compatibility status:

Checks if related components are compatible with change 443f37a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-06T08:51:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24584/
CommitID: 443f37a

sarthakaggarwal97 · 2023-09-06T09:23:43Z

@reta @andrross build is green, please check this out.

reta · 2023-09-06T11:48:07Z

@reta @andrross build is green, please check this out.

Thanks @sarthakaggarwal97 , still LGTM to me :)

sarthakaggarwal97 · 2023-09-06T11:55:28Z

@reta can we merge, and make it in time with for 2.10 release?

reta · 2023-09-06T11:58:59Z

@reta can we merge, and make it in time with for 2.10 release?

We need @andrross signoff as well

Signed-off-by: Andrew Ross <[email protected]>

reta · 2023-09-06T16:06:43Z

💯 ready :-) thanks @andrross & @sarthakaggarwal97 !

github-actions · 2023-09-06T16:32:34Z

Compatibility status:

Checks if related components are compatible with change 5e92890

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-06T16:53:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites

URL: https://build.ci.opensearch.org/job/gradle-check/24676/
CommitID: 5e92890
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-09-06T16:55:00Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9658-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 76f1b52647cf19bd19e3ce9d0759953b5d376593
# Push it to GitHub
git push --set-upstream origin backport/backport-9658-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9658-to-2.x.

reta · 2023-09-06T16:55:52Z

@sarthakaggarwal97 could you please take 2.x backport ? thank you

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> (cherry picked from commit 76f1b52)

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> (cherry picked from commit 76f1b52) Signed-off-by: Andrew Ross <[email protected]>

* introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>

* introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> (cherry picked from commit 90310c3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) (cherry picked from commit 90310c3) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…pensearch-project#9832) (opensearch-project#9844) * introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) (cherry picked from commit 90310c3) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

sarthakaggarwal97 force-pushed the pluginize-zstd branch from 4218b0e to 3d2835b Compare August 31, 2023 09:07

sarthakaggarwal97 force-pushed the pluginize-zstd branch from 3d2835b to e9dc084 Compare August 31, 2023 18:09

sarthakaggarwal97 force-pushed the pluginize-zstd branch 2 times, most recently from 4ff3aac to 0d0aea6 Compare August 31, 2023 18:43

sarthakaggarwal97 force-pushed the pluginize-zstd branch from 42fdd85 to 8bd83a6 Compare August 31, 2023 19:05

sarthakaggarwal97 force-pushed the pluginize-zstd branch from 8bd83a6 to f54a80e Compare August 31, 2023 19:24

sarthakaggarwal97 force-pushed the pluginize-zstd branch from f54a80e to a9512b5 Compare August 31, 2023 19:33

sarthakaggarwal97 marked this pull request as ready for review August 31, 2023 20:09

sarthakaggarwal97 requested review from reta, anasalkouz, andrross and Bukhtawar as code owners August 31, 2023 20:09

andrross approved these changes Sep 6, 2023

View reviewed changes

Merge branch 'main' into pluginize-zstd

5e92890

Signed-off-by: Andrew Ross <[email protected]>

reta merged commit 76f1b52 into opensearch-project:main Sep 6, 2023

reta added backport 2.x Backport to 2.x branch v2.10.0 labels Sep 6, 2023

opensearch-trigger-bot bot added the backport-failed label Sep 6, 2023

andrross mentioned this pull request Sep 6, 2023

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

Merged

sarthakaggarwal97 mentioned this pull request Sep 11, 2023

[DOC] ZStd Documentation update to exclude knn and correlation indices for v2.10 opensearch-project/documentation-website#5002

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing ZStd compression codec plugin #9658

Introducing ZStd compression codec plugin #9658

sarthakaggarwal97 commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

sarthakaggarwal97 commented Sep 6, 2023

reta commented Sep 6, 2023

sarthakaggarwal97 commented Sep 6, 2023

reta commented Sep 6, 2023 •

edited

Loading

reta commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

opensearch-trigger-bot bot commented Sep 6, 2023

reta commented Sep 6, 2023

Introducing ZStd compression codec plugin #9658

Introducing ZStd compression codec plugin #9658

Conversation

sarthakaggarwal97 commented Aug 31, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

sarthakaggarwal97 commented Sep 6, 2023

reta commented Sep 6, 2023

sarthakaggarwal97 commented Sep 6, 2023

sarthakaggarwal97 commented Aug 31, 2023 •

edited

Loading

reta commented Sep 6, 2023 •

edited

Loading

reta commented Sep 6, 2023 •

edited

Loading