Merge remote-tracking branch 'origin/main' into clean-extension-manager

Signed-off-by: Peter Nied <[email protected]>

.ci/bwcVersions

@@ -17,4 +17,5 @@ BWC_VERSION:
   - "2.6.0"
   - "2.6.1"
   - "2.7.0"
+  - "2.7.1"
   - "2.8.0"

CHANGELOG.md

@@ -88,6 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [Extensions] Moving Extensions APIs to protobuf serialization. ([#6960](https://github.com/opensearch-project/OpenSearch/pull/6960))
 - [Extensions] Add IdentityPlugin into core to support Extension identities ([#7246](https://github.com/opensearch-project/OpenSearch/pull/7246))
 - Add descending order search optimization through reverse segment read. ([#7244](https://github.com/opensearch-project/OpenSearch/pull/7244))
+- [Extensions] Moving RestActions APIs to protobuf serialization. ([#7302](https://github.com/opensearch-project/OpenSearch/pull/7302))
 
 ### Dependencies
 - Bump `jackson` from 2.14.2 to 2.15.0 ([#7286](https://github.com/opensearch-project/OpenSearch/pull/7286)

DEVELOPER_GUIDE.md

@@ -26,6 +26,7 @@
     - [`server`](#server)
     - [`test`](#test)
   - [Java Language Formatting Guidelines](#java-language-formatting-guidelines)
+  - [Adding Dependencies](#adding-dependencies)
     - [Editor / IDE Support](#editor--ide-support)
     - [Formatting Failures](#formatting-failures)
   - [Gradle Build](#gradle-build)
@@ -39,7 +40,7 @@
       - [Distribution Download Plugin](#distribution-download-plugin)
     - [Creating fat-JAR of a Module](#creating-fat-jar-of-a-module)
   - [Components](#components)
-    - [Build libraries & interfaces](#build-libraries--interfaces)
+    - [Build Libraries & Interfaces](#build-libraries--interfaces)
     - [Clients & Libraries](#clients--libraries)
     - [Plugins](#plugins-1)
     - [Indexing & Search](#indexing--search)
@@ -328,6 +329,18 @@ Please follow these formatting guidelines:
 * Note that JavaDoc and block comments i.e. `/* ... */` are not formatted, but line comments i.e `// ...` are.
 * There is an implicit rule that negative boolean expressions should use the form `foo == false` instead of `!foo` for better readability of the code. While this isn't strictly enforced, if might get called out in PR reviews as something to change.
 
+## Adding Dependencies
+
+When adding a new dependency or removing an existing dependency via any `build.gradle` (that are not in the test scope), update the dependency LICENSE and library SHAs.
+
+For example, after adding `api "org.slf4j:slf4j-api:${versions.slf4j}"` to [plugins/discovery-ec2/build.gradle](plugins/discovery-ec2/build.gradle), copy the library `LICENSE.txt` and `NOTICE.txt` to `plugins/discovery-ec2/licenses/slf4j-api-LICENSE.txt` and `plugins/discovery-ec2/licenses/slf4j-api-NOTICE.txt`, then run the following to generate `plugins/discovery-ec2/licenses/slf4j-api-1.7.36.jar.sha1`.
+
+```
+./gradlew :plugins:discovery-ec2:updateSHAs
+```
+
+Ensure that `./gradlew :plugins:discovery-ec2:check` passes before submitting changes.
+
 ### Editor / IDE Support
 
 IntelliJ IDEs can [import](https://blog.jetbrains.com/idea/2014/01/intellij-idea-13-importing-code-formatter-settings-from-eclipse/) the [settings file](buildSrc/formatterConfig.xml), and / or use the [Eclipse Code Formatter](https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter)
@@ -424,7 +437,7 @@ Refer the installed JAR as any other maven artifact, e.g.
 
 As you work in the OpenSearch repo you may notice issues getting labeled with component labels.  It's a housekeeping task to help group together similar pieces of work.  You can pretty much ignore it, but if you're curious, here's what the different labels mean:
 
-### Build libraries & interfaces
+### Build Libraries & Interfaces
 
 Tasks to make sure the build tasks are useful and packaging and distribution are easy.
 
@@ -437,7 +450,6 @@ Includes:
 - Compatibility
 - Javadoc enforcement
 
-
 ### Clients & Libraries
 
 APIs and communication mechanisms for external connections to OpenSearch.  This includes the “library” directory in OpenSearch (a set of common functions).
@@ -499,6 +511,7 @@ Includes:
 Security is our top priority. Avoid checking in credentials.
 
 #### Installation
+
 Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
 ```
 git clone https://github.com/awslabs/git-secrets.git
@@ -507,6 +520,7 @@ make install
 ```
 
 #### Configuration
+
 You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
 ```
 git secrets --install
@@ -533,11 +547,13 @@ OpenSearch addresses backward and forward compatibility at three different level
 to ensure backwards compatibility are provided below.
 
 #### Data
+
 The data level consists of index and application data file formats. OpenSearch guarantees file formats and indexes are compatible only back to the first release of
 the previous major version. If on disk formats or encodings need to be changed (including index data, cluster state, or any other persisted data) developers must
 use Version checks accordingly (e.g., `Version.onOrAfter`, `Version.before`) to guarantee backwards compatibility.
 
 #### Developer API
+
 The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes
 obvious components such as the Plugin framework and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is important
 to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with `@opensearch.api`
@@ -547,6 +563,7 @@ guarantee backwards compatibility and may change at any time. The `@deprecated`
 either changed or planned to be removed across minor versions.
 
 #### User API
+
 The User API consists of integration specifications (e.g., [Query Domain Specific Language](https://opensearch.org/docs/latest/opensearch/query-dsl/index/),
 [field mappings](https://opensearch.org/docs/latest/opensearch/mappings/)) and endpoints (e.g., [`_search`](https://opensearch.org/docs/latest/api-reference/search/),
 [`_cat`](https://opensearch.org/docs/latest/api-reference/cat/index/)) users rely on to integrate and use OpenSearch. Backwards compatibility is critical to the
@@ -556,6 +573,7 @@ users of any changes by adding the `>breaking` label on Pull Requests, adding an
 and a log message to the OpenSearch deprecation log files using the `DeprecationLogger`.
 
 #### Experimental Development
+
 Rapidly developing new features often benefit from several release cycles before committing to an official and long term supported (LTS) API. To enable this cycle OpenSearch
 uses an Experimental Development process leveraging [Feature Flags](https://featureflags.io/feature-flags/). This allows a feature to be developed using the same process as
 a LTS feature but with additional guard rails and communication mechanisms to signal to the users and development community the feature is not yet stable, may change in a future

libs/core/src/main/java/org/opensearch/Version.java

@@ -86,6 +86,7 @@ public class Version implements Comparable<Version>, ToXContentFragment {
     public static final Version V_2_6_0 = new Version(2060099, org.apache.lucene.util.Version.LUCENE_9_5_0);
     public static final Version V_2_6_1 = new Version(2060199, org.apache.lucene.util.Version.LUCENE_9_5_0);
     public static final Version V_2_7_0 = new Version(2070099, org.apache.lucene.util.Version.LUCENE_9_5_0);
+    public static final Version V_2_7_1 = new Version(2070199, org.apache.lucene.util.Version.LUCENE_9_5_0);
     public static final Version V_2_8_0 = new Version(2080099, org.apache.lucene.util.Version.LUCENE_9_5_0);
     public static final Version V_3_0_0 = new Version(3000099, org.apache.lucene.util.Version.LUCENE_9_6_0);
     public static final Version CURRENT = V_3_0_0;

server/build.gradle

@@ -353,6 +353,10 @@ tasks.named("missingJavadoc").configure {
    */
   dependsOn("generateProto")
   javadocMissingIgnore = [
+    "org.opensearch.extensions.proto.ExtensionIdentityProto",
+    "org.opensearch.extensions.proto.ExtensionIdentityProto.ExtensionIdentityOrBuilder",
+    "org.opensearch.extensions.proto.RegisterRestActionsProto",
+    "org.opensearch.extensions.proto.RegisterRestActionsProto.RegisterRestActionsOrBuilder",
     "org.opensearch.extensions.proto.ExtensionRequestProto",
     "org.opensearch.extensions.proto.ExtensionRequestProto.ExtensionRequestOrBuilder",
     "org.opensearch.extensions.proto"

server/src/internalClusterTest/java/org/opensearch/action/admin/cluster/stats/ClusterStatsIT.java

@@ -37,6 +37,7 @@
 import org.opensearch.action.admin.cluster.node.stats.NodeStats;
 import org.opensearch.action.admin.cluster.node.stats.NodesStatsRequest;
 import org.opensearch.action.admin.cluster.node.stats.NodesStatsResponse;
+import org.opensearch.client.Client;
 import org.opensearch.client.Requests;
 import org.opensearch.cluster.health.ClusterHealthStatus;
 import org.opensearch.cluster.node.DiscoveryNodeRole;
@@ -152,11 +153,12 @@ public void testNodeCountsWithDeprecatedMasterRole() throws ExecutionException,
 
         Map<String, Integer> expectedCounts = getExpectedCounts(0, 1, 1, 0, 0, 0, 0);
 
-        ClusterStatsResponse response = client().admin().cluster().prepareClusterStats().get();
+        Client client = client();
+        ClusterStatsResponse response = client.admin().cluster().prepareClusterStats().get();
         assertCounts(response.getNodesStats().getCounts(), total, expectedCounts);
 
         Set<String> expectedRoles = Set.of(DiscoveryNodeRole.MASTER_ROLE.roleName());
-        assertEquals(expectedRoles, getNodeRoles(0));
+        assertEquals(expectedRoles, getNodeRoles(client, 0));
     }
 
     private static void incrementCountForRole(String role, Map<String, Integer> counts) {
@@ -327,14 +329,15 @@ public void testNodeRolesWithMasterLegacySettings() throws ExecutionException, I
 
         Map<String, Integer> expectedCounts = getExpectedCounts(0, 1, 1, 0, 1, 0, 0);
 
-        ClusterStatsResponse clusterStatsResponse = client().admin().cluster().prepareClusterStats().get();
+        Client client = client();
+        ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get();
         assertCounts(clusterStatsResponse.getNodesStats().getCounts(), total, expectedCounts);
 
         Set<String> expectedRoles = Set.of(
             DiscoveryNodeRole.MASTER_ROLE.roleName(),
             DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE.roleName()
         );
-        assertEquals(expectedRoles, getNodeRoles(0));
+        assertEquals(expectedRoles, getNodeRoles(client, 0));
     }
 
     public void testNodeRolesWithClusterManagerRole() throws ExecutionException, InterruptedException {
@@ -356,14 +359,15 @@ public void testNodeRolesWithClusterManagerRole() throws ExecutionException, Int
 
         Map<String, Integer> expectedCounts = getExpectedCounts(0, 1, 1, 0, 1, 0, 0);
 
-        ClusterStatsResponse clusterStatsResponse = client().admin().cluster().prepareClusterStats().get();
+        Client client = client();
+        ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get();
         assertCounts(clusterStatsResponse.getNodesStats().getCounts(), total, expectedCounts);
 
         Set<String> expectedRoles = Set.of(
             DiscoveryNodeRole.CLUSTER_MANAGER_ROLE.roleName(),
             DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE.roleName()
         );
-        assertEquals(expectedRoles, getNodeRoles(0));
+        assertEquals(expectedRoles, getNodeRoles(client, 0));
     }
 
     public void testNodeRolesWithSeedDataNodeLegacySettings() throws ExecutionException, InterruptedException {
@@ -379,15 +383,16 @@ public void testNodeRolesWithSeedDataNodeLegacySettings() throws ExecutionExcept
 
         Map<String, Integer> expectedRoleCounts = getExpectedCounts(1, 1, 1, 0, 1, 0, 0);
 
-        ClusterStatsResponse clusterStatsResponse = client().admin().cluster().prepareClusterStats().get();
+        Client client = client();
+        ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get();
         assertCounts(clusterStatsResponse.getNodesStats().getCounts(), total, expectedRoleCounts);
 
         Set<String> expectedRoles = Set.of(
             DiscoveryNodeRole.MASTER_ROLE.roleName(),
             DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE.roleName(),
             DiscoveryNodeRole.DATA_ROLE.roleName()
         );
-        assertEquals(expectedRoles, getNodeRoles(0));
+        assertEquals(expectedRoles, getNodeRoles(client, 0));
     }
 
     public void testNodeRolesWithDataNodeLegacySettings() throws ExecutionException, InterruptedException {
@@ -405,14 +410,15 @@ public void testNodeRolesWithDataNodeLegacySettings() throws ExecutionException,
 
         Map<String, Integer> expectedRoleCounts = getExpectedCounts(1, 1, 1, 0, 1, 0, 0);
 
-        ClusterStatsResponse clusterStatsResponse = client().admin().cluster().prepareClusterStats().get();
+        Client client = client();
+        ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get();
         assertCounts(clusterStatsResponse.getNodesStats().getCounts(), total, expectedRoleCounts);
 
         Set<Set<String>> expectedNodesRoles = Set.of(
             Set.of(DiscoveryNodeRole.CLUSTER_MANAGER_ROLE.roleName()),
             Set.of(DiscoveryNodeRole.DATA_ROLE.roleName(), DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE.roleName())
         );
-        assertEquals(expectedNodesRoles, Set.of(getNodeRoles(0), getNodeRoles(1)));
+        assertEquals(expectedNodesRoles, Set.of(getNodeRoles(client, 0), getNodeRoles(client, 1)));
     }
 
     private Map<String, Integer> getExpectedCounts(
@@ -435,8 +441,8 @@ private Map<String, Integer> getExpectedCounts(
         return expectedCounts;
     }
 
-    private Set<String> getNodeRoles(int nodeNumber) throws ExecutionException, InterruptedException {
-        NodesStatsResponse nodesStatsResponse = client().admin().cluster().nodesStats(new NodesStatsRequest()).get();
+    private Set<String> getNodeRoles(Client client, int nodeNumber) throws ExecutionException, InterruptedException {
+        NodesStatsResponse nodesStatsResponse = client.admin().cluster().nodesStats(new NodesStatsRequest()).get();
         return nodesStatsResponse.getNodes()
             .get(nodeNumber)
             .getNode()

server/src/main/java/org/opensearch/extensions/ExtensionRequest.java

@@ -13,6 +13,7 @@
 import org.opensearch.common.Nullable;
 import org.opensearch.common.io.stream.StreamInput;
 import org.opensearch.common.io.stream.StreamOutput;
+import org.opensearch.extensions.proto.ExtensionIdentityProto;
 import org.opensearch.extensions.proto.ExtensionRequestProto;
 import org.opensearch.transport.TransportRequest;
 
@@ -35,7 +36,7 @@ public ExtensionRequest(ExtensionRequestProto.RequestType requestType) {
     public ExtensionRequest(ExtensionRequestProto.RequestType requestType, @Nullable String uniqueId) {
         ExtensionRequestProto.ExtensionRequest.Builder builder = ExtensionRequestProto.ExtensionRequest.newBuilder();
         if (uniqueId != null) {
-            builder.setUniqueId(uniqueId);
+            builder.setIdentity(ExtensionIdentityProto.ExtensionIdentity.newBuilder().setUniqueId(uniqueId).build());
         }
         this.request = builder.setRequestType(requestType).build();
     }
@@ -56,24 +57,28 @@ public ExtensionRequestProto.RequestType getRequestType() {
     }
 
     public String getUniqueId() {
-        return request.getUniqueId();
+        return request.getIdentity().getUniqueId();
     }
 
     public String toString() {
         return "ExtensionRequest{" + request.toString() + '}';
     }
 
+    public ExtensionIdentityProto.ExtensionIdentity getExtensionIdentity() {
+        return request.getIdentity();
+    }
+
     @Override
     public boolean equals(Object o) {
         if (this == o) return true;
         if (o == null || getClass() != o.getClass()) return false;
         ExtensionRequest that = (ExtensionRequest) o;
         return Objects.equals(request.getRequestType(), that.request.getRequestType())
-            && Objects.equals(request.getUniqueId(), that.request.getUniqueId());
+            && Objects.equals(request.getIdentity().getUniqueId(), that.request.getIdentity().getUniqueId());
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(request.getRequestType(), request.getUniqueId());
+        return Objects.hash(request.getRequestType(), request.getIdentity().getUniqueId());
     }
 }

server/src/main/java/org/opensearch/extensions/rest/RegisterRestActionsRequest.java

@@ -10,10 +10,11 @@
 
 import org.opensearch.common.io.stream.StreamInput;
 import org.opensearch.common.io.stream.StreamOutput;
+import org.opensearch.extensions.proto.ExtensionIdentityProto.ExtensionIdentity;
+import org.opensearch.extensions.proto.RegisterRestActionsProto.RegisterRestActions;
 import org.opensearch.transport.TransportRequest;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.List;
 import java.util.Objects;
 
@@ -23,51 +24,48 @@
  * @opensearch.internal
  */
 public class RegisterRestActionsRequest extends TransportRequest {
-    private String uniqueId;
-    private List<String> restActions;
-    private List<String> deprecatedRestActions;
+    private final RegisterRestActions request;
 
     public RegisterRestActionsRequest(String uniqueId, List<String> restActions, List<String> deprecatedRestActions) {
-        this.uniqueId = uniqueId;
-        this.restActions = new ArrayList<>(restActions);
-        this.deprecatedRestActions = new ArrayList<>(deprecatedRestActions);
+        ExtensionIdentity identity = ExtensionIdentity.newBuilder().setUniqueId(uniqueId).build();
+        this.request = RegisterRestActions.newBuilder()
+            .setIdentity(identity)
+            .addAllRestActions(restActions)
+            .addAllDeprecatedRestActions(deprecatedRestActions)
+            .build();
     }
 
     public RegisterRestActionsRequest(StreamInput in) throws IOException {
         super(in);
-        uniqueId = in.readString();
-        restActions = in.readStringList();
-        deprecatedRestActions = in.readStringList();
+        request = RegisterRestActions.parseFrom(in.readByteArray());
     }
 
     @Override
     public void writeTo(StreamOutput out) throws IOException {
         super.writeTo(out);
-        out.writeString(uniqueId);
-        out.writeStringCollection(restActions);
-        out.writeStringCollection(deprecatedRestActions);
+        out.writeByteArray(request.toByteArray());
     }
 
     public String getUniqueId() {
-        return uniqueId;
+        return request.getIdentity().getUniqueId();
     }
 
     public List<String> getRestActions() {
-        return List.copyOf(restActions);
+        return List.copyOf(request.getRestActionsList());
     }
 
     public List<String> getDeprecatedRestActions() {
-        return List.copyOf(deprecatedRestActions);
+        return List.copyOf(request.getDeprecatedRestActionsList());
     }
 
     @Override
     public String toString() {
-        return "RestActionsRequest{uniqueId="
-            + uniqueId
+        return "RestActionsRequest{Identity="
+            + request.getIdentity()
             + ", restActions="
-            + restActions
+            + request.getRestActionsList()
             + ", deprecatedRestActions="
-            + deprecatedRestActions
+            + request.getDeprecatedRestActionsList()
             + "}";
     }
 
@@ -76,13 +74,13 @@ public boolean equals(Object obj) {
         if (this == obj) return true;
         if (obj == null || getClass() != obj.getClass()) return false;
         RegisterRestActionsRequest that = (RegisterRestActionsRequest) obj;
-        return Objects.equals(uniqueId, that.uniqueId)
-            && Objects.equals(restActions, that.restActions)
-            && Objects.equals(deprecatedRestActions, that.deprecatedRestActions);
+        return Objects.equals(request.getIdentity().getUniqueId(), that.request.getIdentity().getUniqueId())
+            && Objects.equals(request.getRestActionsList(), that.request.getRestActionsList())
+            && Objects.equals(request.getDeprecatedRestActionsList(), that.request.getDeprecatedRestActionsList());
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(uniqueId, restActions, deprecatedRestActions);
+        return Objects.hash(request.getIdentity(), request.getRestActionsList(), request.getDeprecatedRestActionsList());
     }
 }