Merge branch '2.x' into backport/backport-6917-to-2.x

Signed-off-by: Daniel Widdis <[email protected]>

CHANGELOG.md

@@ -15,29 +15,35 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add wait_for_completion parameter to resize, open, and forcemerge APIs ([#6434](https://github.com/opensearch-project/OpenSearch/pull/6434))
 - [Segment Replication] Apply backpressure when replicas fall behind ([#6563](https://github.com/opensearch-project/OpenSearch/pull/6563))
 - [Remote Store] Integrate remote segment store in peer recovery flow ([#6664](https://github.com/opensearch-project/OpenSearch/pull/6664))
-- [Segment Replication] Add new cluster setting to set replication strategy by default for all indices in cluster. ([#6791](https://github.com/opensearch-project/OpenSearch/pull/6791))
 - Enable sort optimization for all NumericTypes ([#6464](https://github.com/opensearch-project/OpenSearch/pull/6464)
 - Remove 'cluster_manager' role attachment when using 'node.master' deprecated setting ([#6331](https://github.com/opensearch-project/OpenSearch/pull/6331))
+- Add new cluster settings to ignore weighted round-robin routing and fallback to default behaviour. ([#6834](https://github.com/opensearch-project/OpenSearch/pull/6834))
+- Add experimental support for ZSTD compression. ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
+- [Segment Replication] Add point in time and scroll query compatibility. ([#6644](https://github.com/opensearch-project/OpenSearch/pull/6644))
 
 ### Dependencies
 - Bump `org.apache.logging.log4j:log4j-core` from 2.18.0 to 2.20.0 ([#6490](https://github.com/opensearch-project/OpenSearch/pull/6490))
 - Bump `com.azure:azure-storage-common` from 12.19.3 to 12.20.0 ([#6492](https://github.com/opensearch-project/OpenSearch/pull/6492)
-- Bump `com.azure:azure-storage-blob` from 12.15.0 to 12.20.0 ([#6921](https://github.com/opensearch-project/OpenSearch/pull/6921)
 - Bump `snakeyaml` from 1.33 to 2.0 ([#6511](https://github.com/opensearch-project/OpenSearch/pull/6511))
 - Bump `io.projectreactor.netty:reactor-netty` from 1.1.3 to 1.1.4
 - Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.15.2 to 0.16.11
 - Bump `net.minidev:json-smart` from 2.4.8 to 2.4.9
-- Bump `com.google.protobuf:protobuf-java` from 3.22.0 to 3.22.2
+- Bump `com.google.protobuf:protobuf-java` to 3.22.2 ([#6994](https://github.com/opensearch-project/OpenSearch/pull/6994))
 - Bump Netty to 4.1.90.Final ([#6677](https://github.com/opensearch-project/OpenSearch/pull/6677)
 - Bump `com.diffplug.spotless` from 6.15.0 to 6.17.0
 - Bump `org.apache.zookeeper:zookeeper` from 3.8.0 to 3.8.1
 - Bump `net.minidev:json-smart` from 2.4.7 to 2.4.10
 - Bump `org.apache.maven:maven-model` from 3.6.2 to 3.9.1
 - Bump `org.codehaus.jettison:jettison` from 1.5.3 to 1.5.4 ([#6878](https://github.com/opensearch-project/OpenSearch/pull/6878))
+- Add `com.github.luben:zstd-jni:1.5.4-1` ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
+- Bump: Netty from 4.1.90.Final to 4.1.91.Final , ASM 9.4 to ASM 9.5, ByteBuddy 1.14.2 to 1.14.3 ([#6981](https://github.com/opensearch-project/OpenSearch/pull/6981))
+- Bump `com.azure:azure-storage-blob` from 12.15.0 to 12.21.1
+- Bump `org.gradle.test-retry` from 1.5.1 to 1.5.2
 
 ### Changed
 - Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))
 - [Refactor] XContent base classes from xcontent to core library ([#5902](https://github.com/opensearch-project/OpenSearch/pull/5902))
+- Added a new field type: flat_object ([#6507](https://github.com/opensearch-project/OpenSearch/pull/6507))
 
 ### Deprecated
 - Map, List, and Set in org.opensearch.common.collect ([#6609](https://github.com/opensearch-project/OpenSearch/pull/6609))
@@ -50,6 +56,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fixed bug for searchable snapshot to take 'base_path' of blob into account ([#6558](https://github.com/opensearch-project/OpenSearch/pull/6558))
 - Fix fuzziness validation ([#5805](https://github.com/opensearch-project/OpenSearch/pull/5805))
 - Avoid negative memory result in IndicesQueryCache stats calculation ([#6917](https://github.com/opensearch-project/OpenSearch/pull/6917))
+- Fix GetSnapshots to not return non-existent snapshots with ignore_unavailable=true ([#6839](https://github.com/opensearch-project/OpenSearch/pull/6839))
 
 ### Security
 

benchmarks/src/main/java/org/opensearch/benchmark/store/remote/filecache/FileCacheBenchmark.java

@@ -27,6 +27,8 @@
 import org.openjdk.jmh.annotations.Threads;
 import org.openjdk.jmh.annotations.Warmup;
 import org.openjdk.jmh.infra.Blackhole;
+import org.opensearch.common.breaker.CircuitBreaker;
+import org.opensearch.common.breaker.NoopCircuitBreaker;
 import org.opensearch.index.store.remote.filecache.CachedIndexInput;
 import org.opensearch.index.store.remote.filecache.FileCache;
 import org.opensearch.index.store.remote.filecache.FileCacheFactory;
@@ -91,7 +93,8 @@ public static class CacheParameters {
         public void setup() {
             fileCache = FileCacheFactory.createConcurrentLRUFileCache(
                 (long) maximumNumberOfEntries * INDEX_INPUT.length(),
-                concurrencyLevel
+                concurrencyLevel,
+                new NoopCircuitBreaker(CircuitBreaker.REQUEST)
             );
             for (long i = 0; i < maximumNumberOfEntries; i++) {
                 final Path key = Paths.get(Long.toString(i));

build.gradle

@@ -54,7 +54,7 @@ plugins {
   id 'opensearch.docker-support'
   id 'opensearch.global-build-info'
   id "com.diffplug.spotless" version "6.17.0" apply false
-  id "org.gradle.test-retry" version "1.5.1" apply false
+  id "org.gradle.test-retry" version "1.5.2" apply false
   id "test-report-aggregation"
   id 'jacoco-report-aggregation'
 }

buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java

@@ -119,8 +119,8 @@ public void apply(Project project) {
             TaskProvider<?> depsTask = project.getTasks().register(taskname + "#deps");
             depsTask.configure(t -> t.dependsOn(distribution, examplePlugin));
             depsTasks.put(taskname, depsTask);
-            // TODO - suppressing failure temporarily where duplicate tasks are created for docker.
-            try {
+            // Avoid duplicate tasks such as docker registered in lifecycleTasks
+            if (project.getTasksByName(taskname, false).isEmpty()) {
                 TaskProvider<Test> destructiveTask = configureTestTask(project, taskname, distribution, t -> {
                     t.onlyIf(t2 -> distribution.isDocker() == false || dockerSupport.get().getDockerAvailability().isAvailable);
                     addSysprop(t, DISTRIBUTION_SYSPROP, distribution::getFilepath);
@@ -134,8 +134,6 @@ public void apply(Project project) {
                 }
                 destructiveDistroTest.configure(t -> t.dependsOn(destructiveTask));
                 lifecycleTasks.get(distribution.getType()).configure(t -> t.dependsOn(destructiveTask));
-            } catch (Exception ex) {
-                System.out.println(ex.getMessage());
             }
 
             if ((distribution.getType() == OpenSearchDistribution.Type.DEB || distribution.getType() == OpenSearchDistribution.Type.RPM)

buildSrc/version.properties

@@ -15,7 +15,7 @@ supercsv          = 2.4.0
 # Update to 2.17.2+ is breaking OpenSearchJsonLayout (see https://issues.apache.org/jira/browse/LOG4J2-3562)
 log4j             = 2.17.1
 slf4j             = 1.7.36
-asm               = 9.4
+asm               = 9.5
 jettison          = 1.5.4
 woodstox          = 6.4.0
 kotlin            = 1.7.10
@@ -25,7 +25,7 @@ guava             = 31.1-jre
 # when updating the JNA version, also update the version in buildSrc/build.gradle
 jna               = 5.5.0
 
-netty             = 4.1.90.Final
+netty             = 4.1.91.Final
 joda              = 2.12.2
 
 # client dependencies
@@ -48,7 +48,7 @@ junit             = 4.13.2
 hamcrest          = 2.1
 mockito           = 5.2.0
 objenesis         = 3.2
-bytebuddy         = 1.14.2
+bytebuddy         = 1.14.3
 
 # benchmark dependencies
 jmh               = 1.35

modules/lang-expression/licenses/asm-9.5.jar.sha1

@@ -0,0 +1 @@
+dc6ea1875f4d64fbc85e1691c95b96a3d8569c90

modules/lang-expression/licenses/asm-commons-9.5.jar.sha1

@@ -0,0 +1 @@
+19ab5b5800a3910d30d3a3e64fdb00fd0cb42de0

modules/lang-expression/licenses/asm-tree-9.5.jar.sha1

@@ -0,0 +1 @@
+fd33c8b6373abaa675be407082fdfda35021254a

modules/lang-painless/licenses/asm-9.5.jar.sha1

@@ -0,0 +1 @@
+dc6ea1875f4d64fbc85e1691c95b96a3d8569c90

modules/lang-painless/licenses/asm-analysis-9.5.jar.sha1

@@ -0,0 +1 @@
+490bacc77de7cbc0be1a30bb3471072d705be4a4

modules/lang-painless/licenses/asm-commons-9.5.jar.sha1

@@ -0,0 +1 @@
+19ab5b5800a3910d30d3a3e64fdb00fd0cb42de0

modules/lang-painless/licenses/asm-tree-9.5.jar.sha1

@@ -0,0 +1 @@
+fd33c8b6373abaa675be407082fdfda35021254a

modules/lang-painless/licenses/asm-util-9.5.jar.sha1

@@ -0,0 +1 @@
+64b5a1fc8c1b15ed2efd6a063e976bc8d3dc5ffe

modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/30_search.yml

@@ -482,3 +482,100 @@
                         }]
     - match: { error.root_cause.0.type: "illegal_argument_exception" }
     - match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}
+
+---
+
+"Flat-object fields from within the scripting":
+  - skip:
+      version: " - 2.6.99"
+      reason: "flat_object is introduced in 2.7.0"
+
+  - do:
+      indices.create:
+        index: test
+        body:
+          mappings:
+            properties:
+              flat:
+                type : "flat_object"
+
+  # This document has 6 distinct parts in its flat_object field paths:
+  # - flat.field_1
+  # - flat.field_2
+  # - flat.field_3
+  # - flat.inner
+  # - flat.field_A
+  # - flat.field_B
+  - do:
+      index:
+        index: test
+        id:    1
+        body:  {
+          "flat": {
+            "field_1": "John Doe",
+            "field_2": 33,
+            "field_3": false,
+            "inner": {
+              "field_A": ["foo", "bar"],
+              "field_B": false
+            }
+          }
+        }
+
+  - do:
+      index:
+        index: test
+        id:    2
+        body:  {
+          "flat": {
+            "field_1": "Joe Public",
+            "field_2": 45
+          }
+        }
+
+  - do:
+      indices.refresh:
+        index: test
+
+  # It is possible to filter based on the number of distinct parts of flat_object field paths
+  - do:
+      search:
+        body: {
+          _source: true,
+          query: {
+            bool: {
+              filter: {
+                script: {
+                  script: {
+                    source: "doc['flat'].size() == 6",
+                    lang: "painless"
+                  }
+                }
+              }
+            }
+          }
+        }
+
+  - length:   { hits.hits: 1  }
+  - match: { hits.hits.0._source.flat.field_1: "John Doe" }
+
+  - do:
+      search:
+        body: {
+          _source: true,
+          query: {
+            bool: {
+              filter: {
+                script: {
+                  script: {
+                    source: "doc['flat'].size() < 6",
+                    lang: "painless"
+                  }
+                }
+              }
+            }
+          }
+        }
+
+  - length:   { hits.hits: 1  }
+  - match: { hits.hits.0._source.flat.field_1: "Joe Public" }

modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/reindex/90_remote.yml

@@ -390,14 +390,12 @@
         refresh: true
 
   - do:
-      # sometimes IIS is listening on port 0. In that case we fail in other ways and this test isn't useful.
-      # make sure to stop any local webservers if running this test locally otherwise an s_s_l handshake exception may occur
       catch: /connect_exception|IIS Windows Server/
       reindex:
         body:
           source:
             remote:
-              host: http://127.0.0.1:0
+              host: http://127.0.0.1:54321
             index: source
           dest:
             index: dest

modules/transport-netty4/build.gradle

@@ -252,6 +252,11 @@ thirdPartyAudit {
     'io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField',
     'io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess',
     'io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess',
-    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator'
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator',
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator$1',
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator$2',
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator$3',
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator$4',
+    'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator$5'
   )
 }

modules/transport-netty4/licenses/netty-buffer-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+d8f180291c3501e931968ca7e40ae0323c4eacee

modules/transport-netty4/licenses/netty-codec-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+3044b8e325e33f72c96ac1ea51dda85bef090cc0

modules/transport-netty4/licenses/netty-codec-http-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+4519d2ff470941f0086214b19c9acf992868112f

modules/transport-netty4/licenses/netty-common-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+93e5056462a242718e7689d81180d125c79d7723

modules/transport-netty4/licenses/netty-handler-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+444cf41e4fe28c47ffebba5e77b9458a12f938a1

modules/transport-netty4/licenses/netty-resolver-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+04725d117d4b71ef0e743aa79062489b45472b26

modules/transport-netty4/licenses/netty-transport-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+c2f6bd7143194ca842b535546a405c06aa993934

modules/transport-netty4/licenses/netty-transport-native-unix-common-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+80990b5885b8b67be096d7090cba18f05c67120e

plugins/repository-azure/build.gradle

@@ -54,7 +54,7 @@ dependencies {
   api "io.netty:netty-resolver-dns:${versions.netty}"
   api "io.netty:netty-transport-native-unix-common:${versions.netty}"
   implementation project(':modules:transport-netty4')
-  api 'com.azure:azure-storage-blob:12.20.0'
+  api 'com.azure:azure-storage-blob:12.21.1'
   api 'org.reactivestreams:reactive-streams:1.0.3'
   api 'io.projectreactor:reactor-core:3.4.18'
   api 'io.projectreactor.netty:reactor-netty:1.1.4'

plugins/repository-azure/licenses/azure-storage-blob-12.21.1.jar.sha1

@@ -0,0 +1 @@
+23fc5d2e3266246056a9735315911d200e892d4e

plugins/repository-azure/licenses/netty-codec-dns-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+2c0242c69eee44ee559d02c564dbceee8bf0a5c7

plugins/repository-azure/licenses/netty-codec-http2-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+4ee7027e1653c6ee3f843191e0d932f29e8e14e1

plugins/repository-azure/licenses/netty-codec-socks-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+8f0a52677da411a8ab762c426d723c7f54471504

plugins/repository-azure/licenses/netty-handler-proxy-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+2e3e57eae1a61e4e5f558e39619186fec6c424d3

plugins/repository-azure/licenses/netty-resolver-dns-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+e1567967f5a85a469b10b7394e3e2b90ea5c0b12

plugins/repository-azure/licenses/netty-transport-native-unix-common-4.1.91.Final.jar.sha1

@@ -0,0 +1 @@
+80990b5885b8b67be096d7090cba18f05c67120e