Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dependencies in plugin descriptor properties with semver range #11441

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
7e41183
Add support for dependencies in plugin descriptor properties with sem…
abseth-amzn Dec 4, 2023
7bae459
Remove unused gson licenses
abseth-amzn Dec 5, 2023
8657300
Maintain bwc in PluginInfo with addition of semver range
abseth-amzn Dec 5, 2023
c3b2718
Merge branch 'opensearch-project:main' into semver_range_support
abseth-amzn Dec 15, 2023
d785fa7
Added support for list of ranges
abseth-amzn Dec 14, 2023
52610b8
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Jan 18, 2024
c9c0a08
Add bwc tests and restrict range list size to 1
abseth-amzn Jan 22, 2024
901170d
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Jan 22, 2024
36c6739
Update SemverRange javadoc
abseth-amzn Jan 25, 2024
32bba1f
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Jan 25, 2024
02ef7f7
Minor change to trigger jenkins re-run
abseth-amzn Jan 30, 2024
39d9339
Use jackson instead of gson
abseth-amzn Feb 5, 2024
b917fc1
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Feb 5, 2024
ad4ef52
Remove jackson databind and annotations dependency from server
abseth-amzn Feb 6, 2024
b3161a6
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Feb 6, 2024
51758ea
nit fixes
abseth-amzn Feb 7, 2024
f44551f
Merge remote-tracking branch 'upstream/main' into semver_range_support
abseth-amzn Feb 7, 2024
18a80e1
Minor change to re-run jenkins workflow
abseth-amzn Feb 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- GHA to verify checklist items completion in PR descriptions ([#10800](https://github.com/opensearch-project/OpenSearch/pull/10800))
- Allow to pass the list settings through environment variables (like [], ["a", "b", "c"], ...) ([#10625](https://github.com/opensearch-project/OpenSearch/pull/10625))
- [Admission Control] Integrate CPU AC with ResourceUsageCollector and add CPU AC stats to nodes/stats ([#10887](https://github.com/opensearch-project/OpenSearch/pull/10887))
- Add support for dependencies in plugin descriptor properties with semver range ([#11441](https://github.com/opensearch-project/OpenSearch/pull/11441))
- [S3 Repository] Add setting to control connection count for sync client ([#12028](https://github.com/opensearch-project/OpenSearch/pull/12028))

### Dependencies
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,14 @@ private void printPlugin(Environment env, Terminal terminal, Path plugin, String
PluginInfo info = PluginInfo.readFromProperties(env.pluginsDir().resolve(plugin));
terminal.println(Terminal.Verbosity.SILENT, prefix + info.getName());
terminal.println(Terminal.Verbosity.VERBOSE, info.toString(prefix));
if (info.getOpenSearchVersion().equals(Version.CURRENT) == false) {
if (!PluginsService.isPluginVersionCompatible(info, Version.CURRENT)) {
terminal.errorPrintln(
"WARNING: plugin ["
+ info.getName()
+ "] was built for OpenSearch version "
+ info.getVersion()
+ " but version "
+ info.getOpenSearchVersionRangesString()
+ " and is not compatible with "
+ Version.CURRENT
+ " is required"
);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,10 @@
import org.opensearch.core.util.FileSystemUtils;
import org.opensearch.env.Environment;
import org.opensearch.env.TestEnvironment;
import org.opensearch.semver.SemverRange;
import org.opensearch.test.OpenSearchTestCase;
import org.opensearch.test.PosixPermissionsResetter;
import org.opensearch.test.VersionUtils;
import org.junit.After;
import org.junit.Before;

Expand Down Expand Up @@ -284,6 +286,35 @@ static void writePlugin(String name, Path structure, String... additionalProps)
writeJar(structure.resolve("plugin.jar"), className);
}

static void writePlugin(String name, Path structure, SemverRange opensearchVersionRange, String... additionalProps) throws IOException {
String[] properties = Stream.concat(
Stream.of(
"description",
"fake desc",
"name",
name,
"version",
"1.0",
"dependencies",
"{opensearch:\"" + opensearchVersionRange + "\"}",
"java.version",
System.getProperty("java.specification.version"),
"classname",
"FakePlugin"
),
Arrays.stream(additionalProps)
).toArray(String[]::new);
PluginTestUtil.writePluginProperties(structure, properties);
String className = name.substring(0, 1).toUpperCase(Locale.ENGLISH) + name.substring(1) + "Plugin";
writeJar(structure.resolve("plugin.jar"), className);
}

static Path createPlugin(String name, Path structure, SemverRange opensearchVersionRange, String... additionalProps)
throws IOException {
writePlugin(name, structure, opensearchVersionRange, additionalProps);
return writeZip(structure, null);
}

static void writePluginSecurityPolicy(Path pluginDir, String... permissions) throws IOException {
StringBuilder securityPolicyContent = new StringBuilder("grant {\n ");
for (String permission : permissions) {
Expand Down Expand Up @@ -867,6 +898,32 @@ public void testInstallMisspelledOfficialPlugins() throws Exception {
assertThat(e.getMessage(), containsString("Unknown plugin unknown_plugin"));
}

public void testInstallPluginWithCompatibleDependencies() throws Exception {
Tuple<Path, Environment> env = createEnv(fs, temp);
Path pluginDir = createPluginDir(temp);
String pluginZip = createPlugin("fake", pluginDir, SemverRange.fromString("~" + Version.CURRENT.toString())).toUri()
.toURL()
.toString();
skipJarHellCommand.execute(terminal, Collections.singletonList(pluginZip), false, env.v2());
assertThat(terminal.getOutput(), containsString("100%"));
}

public void testInstallPluginWithIncompatibleDependencies() throws Exception {
Tuple<Path, Environment> env = createEnv(fs, temp);
Path pluginDir = createPluginDir(temp);
// Core version is behind plugin version by one w.r.t patch, hence incompatible
Version coreVersion = Version.CURRENT;
Version pluginVersion = VersionUtils.getVersion(coreVersion.major, coreVersion.minor, (byte) (coreVersion.revision + 1));
String pluginZip = createPlugin("fake", pluginDir, SemverRange.fromString("~" + pluginVersion.toString())).toUri()
.toURL()
.toString();
IllegalArgumentException e = expectThrows(
IllegalArgumentException.class,
() -> skipJarHellCommand.execute(terminal, Collections.singletonList(pluginZip), false, env.v2())
);
assertThat(e.getMessage(), containsString("Plugin [fake] was built for OpenSearch version ~" + pluginVersion));
}

public void testBatchFlag() throws Exception {
MockTerminal terminal = new MockTerminal();
installPlugin(terminal, true);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -278,12 +278,49 @@ public void testExistingIncompatiblePlugin() throws Exception {
buildFakePlugin(env, "fake desc 2", "fake_plugin2", "org.fake2");

MockTerminal terminal = listPlugins(home);
String message = "plugin [fake_plugin1] was built for OpenSearch version 1.0 but version " + Version.CURRENT + " is required";
String message = "plugin [fake_plugin1] was built for OpenSearch version 5.0.0 and is not compatible with " + Version.CURRENT;
assertEquals("fake_plugin1\nfake_plugin2\n", terminal.getOutput());
assertEquals("WARNING: " + message + "\n", terminal.getErrorOutput());

String[] params = { "-s" };
terminal = listPlugins(home, params);
assertEquals("fake_plugin1\nfake_plugin2\n", terminal.getOutput());
}

public void testPluginWithDependencies() throws Exception {
PluginTestUtil.writePluginProperties(
env.pluginsDir().resolve("fake_plugin1"),
"description",
"fake desc 1",
"name",
"fake_plugin1",
"version",
"1.0",
"dependencies",
"{opensearch:\"" + Version.CURRENT + "\"}",
"java.version",
System.getProperty("java.specification.version"),
"classname",
"org.fake1"
);
String[] params = { "-v" };
MockTerminal terminal = listPlugins(home, params);
assertEquals(
buildMultiline(
"Plugins directory: " + env.pluginsDir(),
"fake_plugin1",
"- Plugin information:",
"Name: fake_plugin1",
"Description: fake desc 1",
"Version: 1.0",
"OpenSearch Version: " + Version.CURRENT.toString(),
"Java Version: " + System.getProperty("java.specification.version"),
"Native Controller: false",
"Extended Plugins: []",
" * Classname: org.fake1",
"Folder name: null"
),
terminal.getOutput()
);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
import org.opensearch.core.concurrency.OpenSearchRejectedExecutionException;
import org.opensearch.core.xcontent.MediaType;
import org.opensearch.core.xcontent.MediaTypeRegistry;
import org.opensearch.semver.SemverRange;

import java.io.ByteArrayInputStream;
import java.io.EOFException;
Expand Down Expand Up @@ -750,6 +751,8 @@ public Object readGenericValue() throws IOException {
return readCollection(StreamInput::readGenericValue, HashSet::new, Collections.emptySet());
case 26:
return readBigInteger();
case 27:
return readSemverRange();
abseth-amzn marked this conversation as resolved.
Show resolved Hide resolved
default:
throw new IOException("Can't read unknown type [" + type + "]");
}
Expand Down Expand Up @@ -1090,6 +1093,10 @@ public Version readVersion() throws IOException {
return Version.fromId(readVInt());
}

public SemverRange readSemverRange() throws IOException {
return SemverRange.fromString(readString());
}

/** Reads the {@link Version} from the input stream */
public Build readBuild() throws IOException {
// the following is new for opensearch: we write the distribution to support any "forks"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
import org.opensearch.core.common.settings.SecureString;
import org.opensearch.core.common.text.Text;
import org.opensearch.core.concurrency.OpenSearchRejectedExecutionException;
import org.opensearch.semver.SemverRange;

import java.io.EOFException;
import java.io.FileNotFoundException;
Expand Down Expand Up @@ -784,6 +785,10 @@ public final void writeOptionalInstant(@Nullable Instant instant) throws IOExcep
o.writeByte((byte) 26);
o.writeString(v.toString());
});
writers.put(SemverRange.class, (o, v) -> {
o.writeByte((byte) 27);
o.writeSemverRange((SemverRange) v);
});
WRITERS = Collections.unmodifiableMap(writers);
}

Expand Down Expand Up @@ -1101,6 +1106,10 @@ public void writeVersion(final Version version) throws IOException {
writeVInt(version.id);
}

public void writeSemverRange(final SemverRange range) throws IOException {
writeString(range.toString());
}

/** Writes the OpenSearch {@link Build} informn to the output stream */
public void writeBuild(final Build build) throws IOException {
// the following is new for opensearch: we write the distribution name to support any "forks" of the code
Expand Down
170 changes: 170 additions & 0 deletions libs/core/src/main/java/org/opensearch/semver/SemverRange.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.semver;

import org.opensearch.Version;
import org.opensearch.common.Nullable;
import org.opensearch.core.xcontent.ToXContentFragment;
import org.opensearch.core.xcontent.XContentBuilder;
import org.opensearch.semver.expr.Caret;
import org.opensearch.semver.expr.Equal;
import org.opensearch.semver.expr.Expression;
import org.opensearch.semver.expr.Tilde;

import java.io.IOException;
import java.util.Objects;
import java.util.Optional;

import static java.util.Arrays.stream;

/**
* Represents a single semver range that allows for specifying which {@code org.opensearch.Version}s satisfy the range.
* It is composed of a range version and a range operator. Following are the supported operators:
* <ul>
* <li>'=' Requires exact match with the range version. For example, =1.2.3 range would match only 1.2.3</li>
* <li>'~' Allows for patch version variability starting from the range version. For example, ~1.2.3 range would match versions greater than or equal to 1.2.3 but less than 1.3.0</li>
* <li>'^' Allows for patch and minor version variability starting from the range version. For example, ^1.2.3 range would match versions greater than or equal to 1.2.3 but less than 2.0.0</li>
* </ul>
*/
public class SemverRange implements ToXContentFragment {
abseth-amzn marked this conversation as resolved.
Show resolved Hide resolved

private final Version rangeVersion;
private final RangeOperator rangeOperator;

public SemverRange(final Version rangeVersion, final RangeOperator rangeOperator) {
this.rangeVersion = rangeVersion;
this.rangeOperator = rangeOperator;
}

/**
* Constructs a {@code SemverRange} from its string representation.
* @param range given range
* @return a {@code SemverRange}
*/
public static SemverRange fromString(final String range) {
RangeOperator rangeOperator = RangeOperator.fromRange(range);
String version = range.replaceFirst(rangeOperator.asEscapedString(), "");
if (!Version.stringHasLength(version)) {
throw new IllegalArgumentException("Version cannot be empty");
}
return new SemverRange(Version.fromString(version), rangeOperator);
}

/**
* Return the range operator for this range.
* @return range operator
*/
public RangeOperator getRangeOperator() {
return rangeOperator;
}

/**
* Return the version for this range.
* @return the range version
*/
public Version getRangeVersion() {
return rangeVersion;

Check warning on line 72 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L72

Added line #L72 was not covered by tests
}

/**
* Check if range is satisfied by given version string.
*
* @param versionToEvaluate version to check
* @return {@code true} if range is satisfied by version, {@code false} otherwise
*/
public boolean isSatisfiedBy(final String versionToEvaluate) {
return isSatisfiedBy(Version.fromString(versionToEvaluate));
}

/**
* Check if range is satisfied by given version.
*
* @param versionToEvaluate version to check
* @return {@code true} if range is satisfied by version, {@code false} otherwise
* @see #isSatisfiedBy(String)
*/
public boolean isSatisfiedBy(final Version versionToEvaluate) {
return this.rangeOperator.expression.evaluate(this.rangeVersion, versionToEvaluate);
}

@Override
public boolean equals(@Nullable final Object o) {
if (this == o) {
return true;

Check warning on line 99 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L99

Added line #L99 was not covered by tests
}
if (o == null || getClass() != o.getClass()) {
return false;

Check warning on line 102 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L102

Added line #L102 was not covered by tests
}
SemverRange range = (SemverRange) o;

Check warning on line 104 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L104

Added line #L104 was not covered by tests
return Objects.equals(rangeVersion, range.rangeVersion) && rangeOperator == range.rangeOperator;
}

@Override
public int hashCode() {
return Objects.hash(rangeVersion, rangeOperator);

Check warning on line 110 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L110

Added line #L110 was not covered by tests
}

@Override
public String toString() {
return rangeOperator.asString() + rangeVersion;
}

@Override
public XContentBuilder toXContent(final XContentBuilder builder, final Params params) throws IOException {
return builder.value(toString());

Check warning on line 120 in libs/core/src/main/java/org/opensearch/semver/SemverRange.java

View check run for this annotation

Codecov / codecov/patch

libs/core/src/main/java/org/opensearch/semver/SemverRange.java#L120

Added line #L120 was not covered by tests
}

/**
* A range operator.
*/
public enum RangeOperator {

EQ("=", new Equal()),
TILDE("~", new Tilde()),
CARET("^", new Caret()),
DEFAULT("", new Equal());

private final String operator;
private final Expression expression;

RangeOperator(final String operator, final Expression expression) {
this.operator = operator;
this.expression = expression;
}

/**
* String representation of the range operator.
*
* @return range operator as string
*/
public String asString() {
return operator;
}

/**
* Escaped string representation of the range operator,
* if operator is a regex character.
*
* @return range operator as escaped string, if operator is a regex character
*/
public String asEscapedString() {
if (Objects.equals(operator, "^")) {
return "\\^";
}
return operator;
}

public static RangeOperator fromRange(final String range) {
Optional<RangeOperator> rangeOperator = stream(values()).filter(
operator -> operator != DEFAULT && range.startsWith(operator.asString())
).findFirst();
return rangeOperator.orElse(DEFAULT);
}
}
}
Loading
Loading