Initial search pipelines implementation #6587

msfroh · 2023-03-08T21:14:14Z

Description

This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80).

Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses.

With this commit we can:

Can create, retrieve, update, and delete search pipelines.
Transform search requests and responses by explicitly referencing a pipeline.

Later work will include:

Adding an index setting to specify a default search pipeline.
Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline).
Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!)

Issues Resolved

opensearch-project/search-processor#97

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-03-08T21:21:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12130/
CommitID: 41e4ae0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-08T21:32:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12131/
CommitID: daf8a19
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-08T21:42:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12132/
CommitID: fd0715e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-08T22:56:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12133/
CommitID: 782f90f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-04T22:04:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13433/
CommitID: 13a0419
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta · 2023-04-05T13:34:20Z

server/src/test/java/org/opensearch/search/pipeline/SearchPipelineServiceTests.java

 public class SearchPipelineServiceTests extends OpenSearchTestCase {
+    @BeforeClass
+    public static void enableFeature() {


The Gradle runs test in parallel but the system properties (changed from tests) introduce instability since they affect the whole forked JVM that could run a few test suites simultaneously. Could we refactor the test case to use settings instead? Thank you.

Do you know if the parallel tests run in their own classloader?

Looking at how we've implemented FeatureFlags, it looks like the settings are held in a static variable, so using FeatureFlags.initializeSettings would similarly pollute things across all tests running in the same classloader.

For testing purposes (since it's just this test), I think I might add a boolean field to SearchPipelineService that bypasses the FeatureFlags for that specific instance.

Do you know if the parallel tests run in their own classloader?

I think we definitely fork the process, but not sure how many forks we create

For testing purposes (since it's just this test), I think I might add a boolean field to SearchPipelineService that bypasses the FeatureFlags for that specific instance.

Probably cleaner (and safer) so feature flag won't spread around, thank you

- Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <[email protected]>

reta · 2023-04-05T17:34:53Z

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java

@@ -426,4 +428,12 @@ static class PipelineHolder {
            this.pipeline = Objects.requireNonNull(pipeline);
        }
    }
+
+    private boolean isFeatureEnabled() {


@msfroh since feature flags are on JVM level (they are not dynamic), I think we could use just simple enabled property that will be passed to constructor (as FeatureFlags.isEnabled(FeatureFlags.SEARCH_PIPELINE) from core or manually during testing), wdyt?

github-actions · 2023-04-05T17:37:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13550/
CommitID: 8e1ba27
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-05T17:37:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13551/
CommitID: 179e4aa
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

@reta

Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <[email protected]>

reta · 2023-04-05T18:11:39Z

server/src/main/java/org/opensearch/node/Node.java

@@ -979,7 +980,8 @@ protected Node(
                xContentRegistry,
                namedWriteableRegistry,
                pluginsService.filterPlugins(SearchPipelinePlugin.class),
-                client
+                client,
+                FeatureFlags.isEnabled(SEARCH_PIPELINE)


github-actions · 2023-04-05T18:26:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13554/
CommitID: f49542c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

andrross · 2023-04-05T18:37:21Z

server/src/main/java/org/opensearch/action/ActionModule.java

+        registerHandler.accept(new RestPutSearchPipelineAction());
+        registerHandler.accept(new RestGetSearchPipelineAction());
+        registerHandler.accept(new RestDeleteSearchPipelineAction());


How about guarding these registrations with the feature flag similar to how REMOTE_STORE does it below?

Signed-off-by: Michael Froh <[email protected]>

github-actions · 2023-04-05T19:15:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollWithOngoingSegmentReplication

URL: https://build.ci.opensearch.org/job/gradle-check/13557/
CommitID: 402e8b6
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

navneet1v · 2023-04-08T16:08:04Z

@andrross @msfroh can someone merge this code as it is approved

noCharger · 2023-04-10T17:59:28Z

@andrross @msfroh can someone merge this code as it is approved

+1

opensearch-trigger-bot · 2023-04-10T18:05:37Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6587-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ee990bd40ceacbf9ebd6ddea0874aa98c48ece47
# Push it to GitHub
git push --set-upstream origin backport/backport-6587-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6587-to-2.x.

andrross · 2023-04-10T18:44:38Z

@msfroh Can you look into creating the backport since it couldn't be automatically cherry-picked?

msfroh · 2023-04-10T20:16:46Z

Can you look into creating the backport since it couldn't be automatically cherry-picked?

On it, thanks! 👍

@reta

* Initial search pipelines implementation This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80). Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses. With this commit we can: 1. Can create, retrieve, update, and delete search pipelines. 2. Transform search requests and responses by explicitly referencing a pipeline. Later work will include: 1. Adding an index setting to specify a default search pipeline. 2. Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline). 3. Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!) Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta and @navneet1v 1. SearchPipelinesClient: JavaDoc fix 2. SearchRequest: Check versions when (de)serializing new "pipeline" property. 3. Rename SearchPipelinesPlugin -> SearchPipelinePlugin. 4. Pipeline: Change visibility to package private 5. SearchPipelineProcessingException: New exception type to wrap exceptions thrown when executing a pipeline. Bonus: Added an integration test for filter_query request processor. Signed-off-by: Michael Froh <[email protected]> * Register SearchPipelineProcessingException Also added more useful messages to unit tests to explicitly explain what hoops need to be jumped through in order to add a new serializable exception. Signed-off-by: Michael Froh <[email protected]> * Remove unneeded dependencies from search-pipeline-common I had copied some dependencies from ingest-common, but they are not used by search-pipeline-common (yet). Signed-off-by: Michael Froh <[email protected]> * Avoid cloning SearchRequest if no SearchRequestProcessors Also, add tests to confirm that a pipeline with no processors works fine (as a no-op). Signed-off-by: Michael Froh <[email protected]> * Use NamedWritableRegistry to deserialize SearchRequest Queries are serialized as NamedWritables, so we need to use a NamedWritableRegistry to deserialize. Signed-off-by: Michael Froh <[email protected]> * Check for empty pipeline with CollectionUtils.isEmpty Signed-off-by: Michael Froh <[email protected]> * Update server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Co-authored-by: Navneet Verma <[email protected]> Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @noCharger Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta - Renamed various classes from "SearchPipelinesSomething" to "SearchPipelineSomething" to be consistent. - Refactored NodeInfo construction in NodeService to avoid ternary operator and improved readability. Signed-off-by: Michael Froh <[email protected]> * Gate search pipelines behind a feature flag Also renamed SearchPipelinesRequestConverters. Signed-off-by: Michael Froh <[email protected]> * More feature flag fixes for search pipeline testing - Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <[email protected]> * Move feature flag into constructor parameter Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <[email protected]> * Move REST handlers behind feature flag Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> Co-authored-by: Navneet Verma <[email protected]> (cherry picked from commit ee990bd)

@reta

* Initial search pipelines implementation (#6587) * Initial search pipelines implementation This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80). Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses. With this commit we can: 1. Can create, retrieve, update, and delete search pipelines. 2. Transform search requests and responses by explicitly referencing a pipeline. Later work will include: 1. Adding an index setting to specify a default search pipeline. 2. Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline). 3. Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!) Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta and @navneet1v 1. SearchPipelinesClient: JavaDoc fix 2. SearchRequest: Check versions when (de)serializing new "pipeline" property. 3. Rename SearchPipelinesPlugin -> SearchPipelinePlugin. 4. Pipeline: Change visibility to package private 5. SearchPipelineProcessingException: New exception type to wrap exceptions thrown when executing a pipeline. Bonus: Added an integration test for filter_query request processor. Signed-off-by: Michael Froh <[email protected]> * Register SearchPipelineProcessingException Also added more useful messages to unit tests to explicitly explain what hoops need to be jumped through in order to add a new serializable exception. Signed-off-by: Michael Froh <[email protected]> * Remove unneeded dependencies from search-pipeline-common I had copied some dependencies from ingest-common, but they are not used by search-pipeline-common (yet). Signed-off-by: Michael Froh <[email protected]> * Avoid cloning SearchRequest if no SearchRequestProcessors Also, add tests to confirm that a pipeline with no processors works fine (as a no-op). Signed-off-by: Michael Froh <[email protected]> * Use NamedWritableRegistry to deserialize SearchRequest Queries are serialized as NamedWritables, so we need to use a NamedWritableRegistry to deserialize. Signed-off-by: Michael Froh <[email protected]> * Check for empty pipeline with CollectionUtils.isEmpty Signed-off-by: Michael Froh <[email protected]> * Update server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Co-authored-by: Navneet Verma <[email protected]> Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @noCharger Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta - Renamed various classes from "SearchPipelinesSomething" to "SearchPipelineSomething" to be consistent. - Refactored NodeInfo construction in NodeService to avoid ternary operator and improved readability. Signed-off-by: Michael Froh <[email protected]> * Gate search pipelines behind a feature flag Also renamed SearchPipelinesRequestConverters. Signed-off-by: Michael Froh <[email protected]> * More feature flag fixes for search pipeline testing - Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <[email protected]> * Move feature flag into constructor parameter Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <[email protected]> * Move REST handlers behind feature flag Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> Co-authored-by: Navneet Verma <[email protected]> (cherry picked from commit ee990bd) * Resolve various backporting issues 1. Can't reference version 3.0.0. 2. Bad merges of adjacent version checks. 3. Use of Apache HTTP client 4 (vs 5). 4. Use of old cluster manager naming in REST params. 5. CollectionUtils didn't have isEmpty for collections. Signed-off-by: Michael Froh <[email protected]> * Support deprecated master_timeout parameter Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]>

@reta

* Initial search pipelines implementation This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80). Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses. With this commit we can: 1. Can create, retrieve, update, and delete search pipelines. 2. Transform search requests and responses by explicitly referencing a pipeline. Later work will include: 1. Adding an index setting to specify a default search pipeline. 2. Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline). 3. Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!) Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta and @navneet1v 1. SearchPipelinesClient: JavaDoc fix 2. SearchRequest: Check versions when (de)serializing new "pipeline" property. 3. Rename SearchPipelinesPlugin -> SearchPipelinePlugin. 4. Pipeline: Change visibility to package private 5. SearchPipelineProcessingException: New exception type to wrap exceptions thrown when executing a pipeline. Bonus: Added an integration test for filter_query request processor. Signed-off-by: Michael Froh <[email protected]> * Register SearchPipelineProcessingException Also added more useful messages to unit tests to explicitly explain what hoops need to be jumped through in order to add a new serializable exception. Signed-off-by: Michael Froh <[email protected]> * Remove unneeded dependencies from search-pipeline-common I had copied some dependencies from ingest-common, but they are not used by search-pipeline-common (yet). Signed-off-by: Michael Froh <[email protected]> * Avoid cloning SearchRequest if no SearchRequestProcessors Also, add tests to confirm that a pipeline with no processors works fine (as a no-op). Signed-off-by: Michael Froh <[email protected]> * Use NamedWritableRegistry to deserialize SearchRequest Queries are serialized as NamedWritables, so we need to use a NamedWritableRegistry to deserialize. Signed-off-by: Michael Froh <[email protected]> * Check for empty pipeline with CollectionUtils.isEmpty Signed-off-by: Michael Froh <[email protected]> * Update server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Co-authored-by: Navneet Verma <[email protected]> Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @noCharger Signed-off-by: Michael Froh <[email protected]> * Incorporate feedback from @reta - Renamed various classes from "SearchPipelinesSomething" to "SearchPipelineSomething" to be consistent. - Refactored NodeInfo construction in NodeService to avoid ternary operator and improved readability. Signed-off-by: Michael Froh <[email protected]> * Gate search pipelines behind a feature flag Also renamed SearchPipelinesRequestConverters. Signed-off-by: Michael Froh <[email protected]> * More feature flag fixes for search pipeline testing - Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <[email protected]> * Move feature flag into constructor parameter Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <[email protected]> * Move REST handlers behind feature flag Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> Co-authored-by: Navneet Verma <[email protected]>

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners March 8, 2023 21:14

msfroh force-pushed the search_pipelines branch from 41e4ae0 to daf8a19 Compare March 8, 2023 21:26

msfroh force-pushed the search_pipelines branch from daf8a19 to fd0715e Compare March 8, 2023 21:39

msfroh force-pushed the search_pipelines branch from fd0715e to 782f90f Compare March 8, 2023 22:37

msfroh force-pushed the search_pipelines branch from 782f90f to b97d3de Compare March 9, 2023 01:13

reta reviewed Apr 5, 2023

View reviewed changes

msfroh added 2 commits April 5, 2023 17:29

More feature flag fixes for search pipeline testing

8e1ba27

- Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <[email protected]>

Merge branch 'main' into search_pipelines

179e4aa

reta reviewed Apr 5, 2023

View reviewed changes

Move feature flag into constructor parameter

f49542c

Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <[email protected]>

reta reviewed Apr 5, 2023

View reviewed changes

reta approved these changes Apr 5, 2023

View reviewed changes

andrross reviewed Apr 5, 2023

View reviewed changes

Move REST handlers behind feature flag

402e8b6

Signed-off-by: Michael Froh <[email protected]>

andrross approved these changes Apr 5, 2023

View reviewed changes

navneet1v mentioned this pull request Apr 6, 2023

[RFC]: Search Phase Results Processor opensearch-project/neural-search#152

Closed

andrross added the backport 2.x Backport to 2.x branch label Apr 10, 2023

andrross merged commit ee990bd into opensearch-project:main Apr 10, 2023

msfroh deleted the search_pipelines branch April 10, 2023 20:16

msfroh mentioned this pull request Apr 13, 2023

[SearchPipelines] Lower compatibility version to 2.7 #7135

Merged

6 tasks

macohen mentioned this pull request Apr 13, 2023

[RFC] Search pipelines opensearch-project/search-processor#80

Closed

1 task

msfroh mentioned this pull request May 15, 2023

[Search Pipelines] Split processor namespace by type (request vs response vs phase) #7576

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial search pipelines implementation #6587

Initial search pipelines implementation #6587

msfroh commented Mar 8, 2023

github-actions bot commented Mar 8, 2023

github-actions bot commented Mar 8, 2023

github-actions bot commented Mar 8, 2023

github-actions bot commented Mar 8, 2023

github-actions bot commented Apr 4, 2023

reta Apr 5, 2023

msfroh Apr 5, 2023

msfroh Apr 5, 2023

reta Apr 5, 2023 •

edited

Loading

reta Apr 5, 2023 •

edited

Loading

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

reta Apr 5, 2023

github-actions bot commented Apr 5, 2023

andrross Apr 5, 2023

github-actions bot commented Apr 5, 2023

navneet1v commented Apr 8, 2023

noCharger commented Apr 10, 2023

opensearch-trigger-bot bot commented Apr 10, 2023

andrross commented Apr 10, 2023

msfroh commented Apr 10, 2023

Initial search pipelines implementation #6587

Initial search pipelines implementation #6587

Conversation

msfroh commented Mar 8, 2023

Description

Issues Resolved

Check List

github-actions bot commented Mar 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

reta Apr 5, 2023

Choose a reason for hiding this comment

msfroh Apr 5, 2023

Choose a reason for hiding this comment

msfroh Apr 5, 2023

Choose a reason for hiding this comment

reta Apr 5, 2023 • edited Loading

Choose a reason for hiding this comment

reta Apr 5, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

reta Apr 5, 2023

Choose a reason for hiding this comment

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

andrross Apr 5, 2023

Choose a reason for hiding this comment

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

navneet1v commented Apr 8, 2023

noCharger commented Apr 10, 2023

opensearch-trigger-bot bot commented Apr 10, 2023

andrross commented Apr 10, 2023

msfroh commented Apr 10, 2023

reta Apr 5, 2023 •

edited

Loading

reta Apr 5, 2023 •

edited

Loading