Skip to content

Commit

Permalink
Merge branch '2.x' into backport/backport-6875-to-2.x
Browse files Browse the repository at this point in the history
Signed-off-by: bansvaru <[email protected]>
  • Loading branch information
linuxpi committed May 19, 2023
2 parents 90b6649 + 77f67db commit 9bb878a
Show file tree
Hide file tree
Showing 203 changed files with 3,710 additions and 1,465 deletions.
16 changes: 14 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- [Extensions] Moving Extensions APIs to support cross versions via protobuf. ([#7402](https://github.com/opensearch-project/OpenSearch/issues/7402))
- [Extensions] Add IdentityPlugin into core to support Extension identities ([#7246](https://github.com/opensearch-project/OpenSearch/pull/7246))
- Add connectToNodeAsExtension in TransportService ([#6866](https://github.com/opensearch-project/OpenSearch/pull/6866))
- [Search Pipelines] Accept pipelines defined in search source ([#7253](https://github.com/opensearch-project/OpenSearch/pull/7253))
- Add descending order search optimization through reverse segment read. ([#7244](https://github.com/opensearch-project/OpenSearch/pull/7244))
- Add 'unsigned_long' numeric field type ([#6237](https://github.com/opensearch-project/OpenSearch/pull/6237))
- Add back primary shard preference for queries ([#7375](https://github.com/opensearch-project/OpenSearch/pull/7375))
- Add descending order search optimization through reverse segment read. ([#7244](https://github.com/opensearch-project/OpenSearch/pull/7244))
- Adds ExtensionsManager.lookupExtensionSettingsById ([#7466](https://github.com/opensearch-project/OpenSearch/pull/7466))
- Provide mechanism to configure XContent parsing constraints (after update to Jackson 2.15.0 and above) ([#7550](https://github.com/opensearch-project/OpenSearch/pull/7550))
- Support to clear filecache using clear indices cache API ([#7498](https://github.com/opensearch-project/OpenSearch/pull/7498))

### Dependencies
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.0.0 to 12.1.0
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.0.0 to 12.1.3 (#7564)
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.2.0
- Bump `com.google.protobuf:protobuf-java` from 3.22.2 to 3.22.3
- Bump `jackson` from 2.14.2 to 2.15.0 ([#7286](https://github.com/opensearch-project/OpenSearch/pull/7286))
Expand All @@ -25,10 +30,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `org.apache.shiro:shiro-core` from 1.9.1 to 1.11.0 ([#7397](https://github.com/opensearch-project/OpenSearch/pull/7397))
- Bump `jetty-server` in hdfs-fixture from 9.4.49.v20220914 to 9.4.51.v20230217 ([#7405](https://github.com/opensearch-project/OpenSearch/pull/7405))
- OpenJDK Update (April 2023 Patch releases) ([#7448](https://github.com/opensearch-project/OpenSearch/pull/7448)
- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0 (#7462)
- Bump `com.azure:azure-core` from 1.34.0 to 1.39.0
- Bump `com.networknt:json-schema-validator` from 1.0.78 to 1.0.81 (#7460)
- Bump Apache Lucene to 9.6.0 ([#7505](https://github.com/opensearch-project/OpenSearch/pull/7505))
- Bump `com.google.cloud:google-cloud-core-http` from 1.93.3 to 2.17.0 (#7488)
- Bump `com.google.guava:guava` from 30.1.1-jre to 31.1-jre (#7565)

### Changed
- Enable `./gradlew build` on MacOS by disabling bcw tests ([#7303](https://github.com/opensearch-project/OpenSearch/pull/7303))
- Moved concurrent-search from sandbox plugin to server module behind feature flag ([#7203](https://github.com/opensearch-project/OpenSearch/pull/7203))
- Allow access to indices cache clear APIs for read only indexes ([#7303](https://github.com/opensearch-project/OpenSearch/pull/7303))

### Deprecated

Expand All @@ -39,4 +51,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

### Security

[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.7...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.7...2.x
4 changes: 2 additions & 2 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ dependencies {
api 'org.apache.ant:ant:1.10.13'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:9.0.0'
api 'com.netflix.nebula:nebula-publishing-plugin:20.3.0'
api 'com.netflix.nebula:gradle-info-plugin:12.1.0'
api 'com.netflix.nebula:gradle-info-plugin:12.1.3'
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.11.0'
api "net.java.dev.jna:jna:5.13.0"
Expand All @@ -118,7 +118,7 @@ dependencies {
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.11'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.1'
api 'com.networknt:json-schema-validator:1.0.73'
api 'com.networknt:json-schema-validator:1.0.81'
api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"

testFixturesApi "junit:junit:${props.getProperty('junit')}"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.gradle.testclusters;

public class ExtensionsProperties {
private String name;
private String uniqueId;
private String hostAddress;
private String port;
private String version;
private String opensearchVersion;
private String minimumCompatibleVersion;

public ExtensionsProperties(
String name,
String uniqueId,
String hostAddress,
String port,
String version,
String opensearchVersion,
String minimumCompatibleVersion
) {
this.name = name;
this.uniqueId = uniqueId;
this.hostAddress = hostAddress;
this.port = port;
this.version = version;
this.opensearchVersion = opensearchVersion;
this.minimumCompatibleVersion = minimumCompatibleVersion;
}

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public String getUniqueId() {
return uniqueId;
}

public void setUniqueId(String uniqueId) {
this.uniqueId = uniqueId;
}

public String getHostAddress() {
return hostAddress;
}

public void setHostAddress(String hostAddress) {
this.hostAddress = hostAddress;
}

public String getPort() {
return port;
}

public void setPort(String port) {
this.port = port;
}

public String getVersion() {
return version;
}

public void setVersion(String version) {
this.version = version;
}

public String getOpensearchVersion() {
return opensearchVersion;
}

public void setOpensearchVersion(String opensearchVersion) {
this.opensearchVersion = opensearchVersion;
}

public String getMinimumCompatibleVersion() {
return minimumCompatibleVersion;
}

public void setMinimumCompatibleVersion(String minimumCompatibleVersion) {
this.minimumCompatibleVersion = minimumCompatibleVersion;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,11 @@ public void setTestDistribution(TestDistribution distribution) {
nodes.all(each -> each.setTestDistribution(distribution));
}

@Override
public void extension(ExtensionsProperties extension) {
nodes.all(each -> each.extension(extension));
}

@Override
public void plugin(Provider<RegularFile> plugin) {
nodes.all(each -> each.plugin(plugin));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@
import org.opensearch.gradle.Version;
import org.opensearch.gradle.VersionProperties;
import org.opensearch.gradle.info.BuildParams;
import org.yaml.snakeyaml.DumperOptions;
import org.yaml.snakeyaml.Yaml;
import org.gradle.api.Action;
import org.gradle.api.Named;
import org.gradle.api.NamedDomainObjectContainer;
Expand Down Expand Up @@ -92,6 +94,7 @@
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Scanner;
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
Expand Down Expand Up @@ -142,6 +145,7 @@ public class OpenSearchNode implements TestClusterConfiguration {
private final Map<String, Configuration> pluginAndModuleConfigurations = new HashMap<>();
private final List<Provider<File>> plugins = new ArrayList<>();
private final List<Provider<File>> modules = new ArrayList<>();
private final List<ExtensionsProperties> extensions = new ArrayList<>();
final LazyPropertyMap<String, CharSequence> settings = new LazyPropertyMap<>("Settings", this);
private final LazyPropertyMap<String, CharSequence> keystoreSettings = new LazyPropertyMap<>("Keystore", this);
private final LazyPropertyMap<String, File> keystoreFiles = new LazyPropertyMap<>("Keystore files", this, FileEntry::new);
Expand Down Expand Up @@ -432,6 +436,42 @@ public void module(String moduleProjectPath) {
module(maybeCreatePluginOrModuleDependency(moduleProjectPath));
}

@Override
public void extension(ExtensionsProperties extensions) {
this.extensions.add(extensions);
}

public void writeExtensionFiles() {
try {
// Creates extensions.yml in the target directory
Path destination = getDistroDir().resolve("extensions").resolve("extensions.yml");
if (!Files.exists(getDistroDir().resolve("extensions"))) {
Files.createDirectory(getDistroDir().resolve("extensions"));
}
DumperOptions dumperOptions = new DumperOptions();
TestExtensionsList extensionsList = new TestExtensionsList(this.extensions);
dumperOptions.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK);
Yaml yaml = new Yaml(dumperOptions);
Files.write(destination, yaml.dump(extensionsList).getBytes());

/*
* SnakeYaml creates a Yaml file with an unnecessary line at the top with the class name
* This section of code removes that line while keeping everything else the same.
*/

Scanner scanner = new Scanner(destination);
scanner.nextLine();
StringBuilder extensionsString = new StringBuilder();
while (scanner.hasNextLine()) {
extensionsString.append("\n" + scanner.nextLine());
}
Files.write(destination, extensionsString.toString().getBytes());

} catch (IOException e) {
throw new UncheckedIOException("Failed to write to extensions.yml", e);
}
}

@Override
public void keystore(String key, String value) {
keystoreSettings.put(key, value);
Expand Down Expand Up @@ -608,7 +648,12 @@ public synchronized void start() {
}
}

if (!extensions.isEmpty()) {
writeExtensionFiles();
}

logToProcessStdout("Creating " + currentConfig.command + " keystore with password set to [" + keystorePassword + "]");

if (keystorePassword.length() > 0) {
runOpenSearchBinScriptWithInput(keystorePassword + "\n" + keystorePassword, currentConfig.keystoreTool, "create", "-p");
} else {
Expand Down Expand Up @@ -892,6 +937,10 @@ private void startOpenSearchProcess() {
environment.clear();
environment.putAll(getOpenSearchEnvironment());

if (!extensions.isEmpty()) {
environment.put("OPENSEARCH_JAVA_OPTS", "-Dopensearch.experimental.feature.extensions.enabled=true");
}

// don't buffer all in memory, make sure we don't block on the default pipes
processBuilder.redirectError(ProcessBuilder.Redirect.appendTo(currentConfig.stderrFile.toFile()));
processBuilder.redirectOutput(ProcessBuilder.Redirect.appendTo(currentConfig.stdoutFile.toFile()));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ public interface TestClusterConfiguration {

void setTestDistribution(TestDistribution distribution);

void extension(ExtensionsProperties extension);

void plugin(Provider<RegularFile> plugin);

void plugin(String pluginProjectPath);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.gradle.testclusters;

import java.util.List;

public class TestExtensionsList {
private List<ExtensionsProperties> extensions;

public TestExtensionsList(List<ExtensionsProperties> extensionsList) {
extensions = extensionsList;
}

public List<ExtensionsProperties> getExtensions() {
return extensions;
}

public void setExtensions(List<ExtensionsProperties> extensionsList) {
extensions = extensionsList;
}
}
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
opensearch = 2.8.0
lucene = 9.5.0
lucene = 9.6.0

bundled_jdk_vendor = adoptium
bundled_jdk = 17.0.7+7
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,7 @@ static Request clearCache(ClearIndicesCacheRequest clearIndicesCacheRequest) {
parameters.withIndicesOptions(clearIndicesCacheRequest.indicesOptions());
parameters.putParam("query", Boolean.toString(clearIndicesCacheRequest.queryCache()));
parameters.putParam("fielddata", Boolean.toString(clearIndicesCacheRequest.fieldDataCache()));
parameters.putParam("file", Boolean.toString(clearIndicesCacheRequest.fileCache()));
parameters.putParam("request", Boolean.toString(clearIndicesCacheRequest.requestCache()));
parameters.putParam("fields", String.join(",", clearIndicesCacheRequest.fields()));
request.addParameters(parameters.asMap());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -596,6 +596,10 @@ public void testClearCache() {
clearIndicesCacheRequest.fields(RequestConvertersTests.randomIndicesNames(1, 5));
expectedParams.put("fields", String.join(",", clearIndicesCacheRequest.fields()));
}
if (OpenSearchTestCase.randomBoolean()) {
clearIndicesCacheRequest.fileCache(OpenSearchTestCase.randomBoolean());
}
expectedParams.put("file", Boolean.toString(clearIndicesCacheRequest.fileCache()));

Request request = IndicesRequestConverters.clearCache(clearIndicesCacheRequest);
StringJoiner endpoint = new StringJoiner("/", "/", "");
Expand Down
5 changes: 4 additions & 1 deletion libs/common/src/main/java/org/opensearch/common/Numbers.java
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,9 @@ public static BigInteger toUnsignedLongExact(Number value) {
private static BigDecimal BIGDECIMAL_GREATER_THAN_LONG_MAX_VALUE = BigDecimal.valueOf(Long.MAX_VALUE).add(BigDecimal.ONE);
private static BigDecimal BIGDECIMAL_LESS_THAN_LONG_MIN_VALUE = BigDecimal.valueOf(Long.MIN_VALUE).subtract(BigDecimal.ONE);
private static BigDecimal BIGDECIMAL_GREATER_THAN_USIGNED_LONG_MAX_VALUE = new BigDecimal(MAX_UNSIGNED_LONG_VALUE).add(BigDecimal.ONE);
private static BigDecimal BIGDECIMAL_LESS_THAN_USIGNED_LONG_MIN_VALUE = new BigDecimal(MIN_UNSIGNED_LONG_VALUE).subtract(
BigDecimal.ONE
);

/** Return the long that {@code stringValue} stores or throws an exception if the
* stored value cannot be converted to a long that stores the exact same
Expand Down Expand Up @@ -212,7 +215,7 @@ public static BigInteger toUnsignedLong(String stringValue, boolean coerce) {
try {
BigDecimal bigDecimalValue = new BigDecimal(stringValue);
if (bigDecimalValue.compareTo(BIGDECIMAL_GREATER_THAN_USIGNED_LONG_MAX_VALUE) >= 0
|| bigDecimalValue.compareTo(BigDecimal.ZERO) <= 0) {
|| bigDecimalValue.compareTo(BIGDECIMAL_LESS_THAN_USIGNED_LONG_MIN_VALUE) <= 0) {
throw new IllegalArgumentException("Value [" + stringValue + "] is out of range for an unsigned long");
}
bigIntegerValue = coerce ? bigDecimalValue.toBigInteger() : bigDecimalValue.toBigIntegerExact();
Expand Down
1 change: 0 additions & 1 deletion libs/core/licenses/lucene-core-9.5.0.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions libs/core/licenses/lucene-core-9.6.0.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2c08c7a491e9d033bb4806e0a45496e3a0667217
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import com.fasterxml.jackson.core.JsonEncoding;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.StreamReadConstraints;
import com.fasterxml.jackson.dataformat.cbor.CBORFactory;
import org.opensearch.core.xcontent.DeprecationHandler;
import org.opensearch.core.xcontent.MediaType;
Expand All @@ -56,6 +57,9 @@
* A CBOR based content implementation using Jackson.
*/
public class CborXContent implements XContent {
public static final int DEFAULT_MAX_STRING_LEN = Integer.parseInt(
System.getProperty("opensearch.xcontent.string.length.max", Integer.toString(Integer.MAX_VALUE) /* no limit */)
);

public static XContentBuilder contentBuilder() throws IOException {
return XContentBuilder.builder(cborXContent);
Expand All @@ -70,6 +74,7 @@ public static XContentBuilder contentBuilder() throws IOException {
// Do not automatically close unclosed objects/arrays in com.fasterxml.jackson.dataformat.cbor.CBORGenerator#close() method
cborFactory.configure(JsonGenerator.Feature.AUTO_CLOSE_JSON_CONTENT, false);
cborFactory.configure(JsonParser.Feature.STRICT_DUPLICATE_DETECTION, true);
cborFactory.setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(DEFAULT_MAX_STRING_LEN).build());
cborXContent = new CborXContent();
}

Expand Down
Loading

0 comments on commit 9bb878a

Please sign in to comment.