Skip to content

Commit

Permalink
Merge branch 'fix-none-tsdb-index-dimension-tests' of github.com:weiz…
Browse files Browse the repository at this point in the history
…ijun/elasticsearch into fix-none-tsdb-index-dimension-tests

* 'fix-none-tsdb-index-dimension-tests' of github.com:weizijun/elasticsearch: (37 commits)
  [docs] Mention JDK 17 in the Contributing docs (elastic#84018)
  Fix GeoIpDownloader startup during rolling upgrade (elastic#84000)
  Script: Fields API for Dense Vector (elastic#83550)
  Move InferenceConfigUpdate under VersionedNamedWriteable (elastic#84022)
  [ML] Fix license feature test cleanup (elastic#84020)
  Replace deprecated api in artifact transforms (elastic#84015)
  QL: Add leniency option to SQL CLI (elastic#83795)
  [Stack Monitoring] add kibana_stats version alias to -mb template (elastic#83930)
  Optimize spliterator for ImmutableOpenMap (elastic#83899)
  Feature usage actions for archive (elastic#83931)
  Use latch to speedup multi feature migration test (elastic#84007)
  Make action names available in NodeClient (elastic#83919)
  [DOCS] Re-add HTTP proxy setings from elastic#82737 (elastic#84001)
  Add CI matrix configuration for snapshot BWC versions (elastic#83990)
  Update YAML Rest tests to check for product header on all responses (elastic#83290)
  TSDB: Add time series aggs cancellation (elastic#83492)
  [DOCS] Fix percolate query headings (elastic#83988)
  [DOCS] Move tip for percolate query example (elastic#83972)
  Simplify LocalExporter cleaner function to fix failing tests (elastic#83812)
  [GCE Discovery] Correcly handle large zones with 500 or more instances (elastic#83785)
  ...
  • Loading branch information
weizijun committed Feb 16, 2022
2 parents f4b06e3 + 8194360 commit 7367932
Show file tree
Hide file tree
Showing 498 changed files with 5,530 additions and 37,849 deletions.
5 changes: 5 additions & 0 deletions .ci/snapshotBwcVersions
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
BWC_VERSION:
- "7.17.1"
- "8.0.1"
- "8.1.0"
- "8.2.0"
10 changes: 5 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ Contributing to the Elasticsearch codebase

**Repository:** [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch)

JDK 16 is required to build Elasticsearch. You must have a JDK 16 installation
JDK 17 is required to build Elasticsearch. You must have a JDK 17 installation
with the environment variable `JAVA_HOME` referencing the path to Java home for
your JDK 16 installation. By default, tests use the same runtime as `JAVA_HOME`.
your JDK 17 installation. By default, tests use the same runtime as `JAVA_HOME`.
However, since Elasticsearch supports JDK 11, the build supports compiling with
JDK 16 and testing on a JDK 11 runtime; to do this, set `RUNTIME_JAVA_HOME`
JDK 17 and testing on a JDK 11 runtime; to do this, set `RUNTIME_JAVA_HOME`
pointing to the Java home of a JDK 11 installation. Note that this mechanism can
be used to test against other JDKs as well, this is not only limited to JDK 11.

Expand Down Expand Up @@ -151,9 +151,9 @@ and then run `curl` in another window like this:
### Importing the project into IntelliJ IDEA

The minimum IntelliJ IDEA version required to import the Elasticsearch project is 2020.1
Elasticsearch builds using Java 16. When importing into IntelliJ you will need
Elasticsearch builds using Java 17. When importing into IntelliJ you will need
to define an appropriate SDK. The convention is that **this SDK should be named
"16"** so that the project import will detect it automatically. For more details
"17"** so that the project import will detect it automatically. For more details
on defining an SDK in IntelliJ please refer to [their documentation](https://www.jetbrains.com/help/idea/sdk.html#define-sdk).
SDK definitions are global, so you can add the JDK from any project, or after
project import. Importing with a missing JDK will still work, IntelliJ will
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This directory contains the microbenchmark suite of Elasticsearch. It relies on
## Purpose

We do not want to microbenchmark everything but the kitchen sink and should typically rely on our
[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with
[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/) with
[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components.
The microbenchmark suite is also handy for ad-hoc microbenchmarks but please remove them again before merging your PR.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1233,6 +1233,11 @@
"total" : 0,
"failed" : 0
}
},
"archive" : {
"available" : false,
"enabled" : true,
"indices_count" : 0
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ protected static void loadBuildSrcVersion(Properties loadedProps, ProviderFactor
);
}
String qualifier = providers.systemProperty("build.version_qualifier")
.forUseAtConfigurationTime()
.getOrElse("");
if (qualifier.isEmpty() == false) {
if (qualifier.matches("(alpha|beta|rc)\\d+") == false) {
Expand All @@ -52,7 +51,6 @@ protected static void loadBuildSrcVersion(Properties loadedProps, ProviderFactor
elasticsearch += "-" + qualifier;
}
final String buildSnapshotSystemProperty = providers.systemProperty("build.snapshot")
.forUseAtConfigurationTime()
.getOrElse("true");
switch (buildSnapshotSystemProperty) {
case "true":
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,6 @@ public void apply(Project project) {
.registerIfAbsent("versions", VersionPropertiesBuildService.class, spec -> {
spec.getParameters().getInfoPath().set(infoPath);
});
project.getExtensions().add("versions", serviceProvider.forUseAtConfigurationTime().get().getProperties());
project.getExtensions().add("versions", serviceProvider.get().getProperties());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ public static int findDefaultParallel(Project project) {
}

private static boolean isMac(ProviderFactory providers) {
return providers.systemProperty("os.name").forUseAtConfigurationTime().getOrElse("").startsWith("Mac");
return providers.systemProperty("os.name").getOrElse("").startsWith("Mac");
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ import org.elasticsearch.gradle.testclusters.StandaloneRestIntegTestTask

def clusterCredentials = [
username: providers.systemProperty('tests.rest.cluster.username')
.forUseAtConfigurationTime()
.getOrElse('test_admin'),
password: providers.systemProperty('tests.rest.cluster.password')
.forUseAtConfigurationTime()
.getOrElse('x-pack-test-password')
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,15 @@

// we do not want any of these dependencies on the compilation classpath
// because they could then be used within Elasticsearch
List<String> FORBIDDEN_DEPENDENCIES = [
'guava'
List<String> FORBIDDEN_DEPENDENCY_GROUPS = [
'com.google.guava'
]

Closure checkDeps = { Configuration configuration ->
configuration.resolutionStrategy.eachDependency {
String artifactName = it.target.name
if (FORBIDDEN_DEPENDENCIES.contains(artifactName)) {
throw new GradleException("Dependency '${artifactName}' on configuration '${configuration.name}' is not allowed. " +
"If it is needed as a transitive depenency, try adding it to the runtime classpath")
if (FORBIDDEN_DEPENDENCY_GROUPS.contains(it.target.group)) {
throw new GradleException("Dependency '${it.target.group}:${it.target.name}' on configuration '${configuration.name}' is not allowed. " +
"If it is needed as a transitive dependency, try adding it to the runtime classpath")
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ tasks.register('configureIdeCheckstyle') {
}

// Applying this stuff, particularly the idea-ext plugin, has a cost so avoid it unless we're running in the IDE
if (providers.systemProperty('idea.active').forUseAtConfigurationTime().getOrNull() == 'true') {
if (providers.systemProperty('idea.active').getOrNull() == 'true') {
project.apply(plugin: org.jetbrains.gradle.ext.IdeaExtPlugin)

tasks.register('configureIdeaGradleJvm') {
Expand Down
6 changes: 3 additions & 3 deletions build-tools-internal/src/main/groovy/elasticsearch.run.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ import org.elasticsearch.gradle.testclusters.RunTask
// apply plugin: 'elasticsearch.internal-testclusters'

testClusters.register("runTask") {
testDistribution = providers.systemProperty('run.distribution').orElse('default').forUseAtConfigurationTime().get()
if (providers.systemProperty('run.distribution').forUseAtConfigurationTime().getOrElse('default') == 'default') {
String licenseType = providers.systemProperty("run.license_type").forUseAtConfigurationTime().getOrElse("basic")
testDistribution = providers.systemProperty('run.distribution').orElse('default').get()
if (providers.systemProperty('run.distribution').getOrElse('default') == 'default') {
String licenseType = providers.systemProperty("run.license_type").getOrElse("basic")
if (licenseType == 'trial') {
setting 'xpack.ml.enabled', 'true'
setting 'xpack.graph.enabled', 'true'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,25 +84,20 @@ public void apply(Project project) {
String remoteRepo = remote.get();
// for testing only we can override the base remote url
String remoteRepoUrl = providerFactory.systemProperty("testRemoteRepo")
.forUseAtConfigurationTime()
.getOrElse("https://github.com/" + remoteRepo + "/elasticsearch.git");
addRemote.setCommandLine(asList("git", "remote", "add", remoteRepo, remoteRepoUrl));
});

TaskProvider<LoggedExec> fetchLatestTaskProvider = tasks.register("fetchLatest", LoggedExec.class, fetchLatest -> {
var gitFetchLatest = project.getProviders()
.systemProperty("tests.bwc.git_fetch_latest")
.forUseAtConfigurationTime()
.orElse("true")
.map(fetchProp -> {
if ("true".equals(fetchProp)) {
return true;
}
if ("false".equals(fetchProp)) {
return false;
}
throw new GradleException("tests.bwc.git_fetch_latest must be [true] or [false] but was [" + fetchProp + "]");
});
var gitFetchLatest = project.getProviders().systemProperty("tests.bwc.git_fetch_latest").orElse("true").map(fetchProp -> {
if ("true".equals(fetchProp)) {
return true;
}
if ("false".equals(fetchProp)) {
return false;
}
throw new GradleException("tests.bwc.git_fetch_latest must be [true] or [false] but was [" + fetchProp + "]");
});
fetchLatest.onlyIf(t -> project.getGradle().getStartParameter().isOffline() == false && gitFetchLatest.get());
fetchLatest.dependsOn(addRemoteTaskProvider);
fetchLatest.setWorkingDir(gitExtension.getCheckoutDir().get());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import java.io.File;

import static org.elasticsearch.gradle.internal.conventions.GUtils.capitalize;
import static org.gradle.api.internal.artifacts.ArtifactAttributes.ARTIFACT_FORMAT;

/**
* Provides a DSL and common configurations to define different types of
Expand Down Expand Up @@ -75,12 +74,14 @@ private void registerAndConfigureDistributionArchivesExtension(Project project)
sub.getArtifacts().add(DEFAULT_CONFIGURATION_NAME, distributionArchive.getArchiveTask());
var extractedConfiguration = sub.getConfigurations().create(EXTRACTED_CONFIGURATION_NAME);
extractedConfiguration.setCanBeResolved(false);
extractedConfiguration.getAttributes().attribute(ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
extractedConfiguration.getAttributes()
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
sub.getArtifacts().add(EXTRACTED_CONFIGURATION_NAME, distributionArchive.getExpandedDistTask());
// The "composite" configuration is specifically used for resolving transformed artifacts in an included build
var compositeConfiguration = sub.getConfigurations().create(COMPOSITE_CONFIGURATION_NAME);
compositeConfiguration.setCanBeResolved(false);
compositeConfiguration.getAttributes().attribute(ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
compositeConfiguration.getAttributes()
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
compositeConfiguration.getAttributes().attribute(Attribute.of("composite", Boolean.class), true);
sub.getArtifacts().add(COMPOSITE_CONFIGURATION_NAME, distributionArchive.getArchiveTask());
sub.getTasks().register("extractedAssemble", task ->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
import org.gradle.api.artifacts.repositories.IvyArtifactRepository;
import org.gradle.api.artifacts.type.ArtifactTypeDefinition;
import org.gradle.api.attributes.Attribute;
import org.gradle.api.internal.artifacts.ArtifactAttributes;

import java.util.Arrays;

Expand All @@ -40,21 +39,21 @@ public void apply(Project project) {
project.getDependencies().getArtifactTypes().maybeCreate(ArtifactTypeDefinition.ZIP_TYPE);
project.getDependencies().registerTransform(UnzipTransform.class, transformSpec -> {
transformSpec.getFrom()
.attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.ZIP_TYPE)
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.ZIP_TYPE)
.attribute(jdkAttribute, true);
transformSpec.getTo()
.attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE)
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE)
.attribute(jdkAttribute, true);
transformSpec.parameters(parameters -> parameters.setTrimmedPrefixPattern(JDK_TRIMMED_PREFIX));
});

ArtifactTypeDefinition tarArtifactTypeDefinition = project.getDependencies().getArtifactTypes().maybeCreate("tar.gz");
project.getDependencies().registerTransform(SymbolicLinkPreservingUntarTransform.class, transformSpec -> {
transformSpec.getFrom()
.attribute(ArtifactAttributes.ARTIFACT_FORMAT, tarArtifactTypeDefinition.getName())
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, tarArtifactTypeDefinition.getName())
.attribute(jdkAttribute, true);
transformSpec.getTo()
.attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE)
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE)
.attribute(jdkAttribute, true);
transformSpec.parameters(parameters -> {
parameters.setTrimmedPrefixPattern(JDK_TRIMMED_PREFIX);
Expand All @@ -65,7 +64,7 @@ public void apply(Project project) {
NamedDomainObjectContainer<Jdk> jdksContainer = project.container(Jdk.class, name -> {
Configuration configuration = project.getConfigurations().create("jdk_" + name);
configuration.setCanBeConsumed(false);
configuration.getAttributes().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
configuration.getAttributes().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
configuration.getAttributes().attribute(jdkAttribute, true);
Jdk jdk = new Jdk(name, configuration, project.getObjects());
configuration.defaultDependencies(dependencies -> {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ private File findRuntimeJavaHome() {
}

private String findJavaHome(String version) {
Provider<String> javaHomeNames = providers.gradleProperty("org.gradle.java.installations.fromEnv").forUseAtConfigurationTime();
Provider<String> javaHomeNames = providers.gradleProperty("org.gradle.java.installations.fromEnv");
String javaHomeEnvVar = getJavaHomeEnvVarName(version);

// Provide a useful error if we're looking for a Java home version that we haven't told Gradle about yet
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
import org.gradle.api.artifacts.Configuration;
import org.gradle.api.artifacts.dsl.DependencyHandler;
import org.gradle.api.artifacts.type.ArtifactTypeDefinition;
import org.gradle.api.internal.artifacts.ArtifactAttributes;
import org.gradle.api.plugins.JavaBasePlugin;
import org.gradle.api.provider.Provider;
import org.gradle.api.specs.Specs;
Expand Down Expand Up @@ -313,7 +312,7 @@ private static Object convertPath(

private static Configuration configureExamplePlugin(Project project) {
Configuration examplePlugin = project.getConfigurations().create(EXAMPLE_PLUGIN_CONFIGURATION);
examplePlugin.getAttributes().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.ZIP_TYPE);
examplePlugin.getAttributes().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.ZIP_TYPE);
DependencyHandler deps = project.getDependencies();
deps.add(EXAMPLE_PLUGIN_CONFIGURATION, deps.project(Map.of("path", ":plugins:analysis-icu", "configuration", "zip")));
return examplePlugin;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,6 @@ public void apply(Project project) {
}

private String systemProperty(String propName) {
return providerFactory.systemProperty(propName).forUseAtConfigurationTime().getOrNull();
return providerFactory.systemProperty(propName).getOrNull();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import org.gradle.api.artifacts.dsl.DependencyHandler;
import org.gradle.api.artifacts.repositories.IvyArtifactRepository;
import org.gradle.api.artifacts.type.ArtifactTypeDefinition;
import org.gradle.api.internal.artifacts.ArtifactAttributes;
import org.gradle.api.model.ObjectFactory;
import org.gradle.api.provider.Property;
import org.gradle.api.provider.Provider;
Expand Down Expand Up @@ -61,14 +60,14 @@ public void setDockerAvailability(Provider<Boolean> dockerAvailability) {
@Override
public void apply(Project project) {
project.getDependencies().registerTransform(UnzipTransform.class, transformSpec -> {
transformSpec.getFrom().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.ZIP_TYPE);
transformSpec.getTo().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
transformSpec.getFrom().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.ZIP_TYPE);
transformSpec.getTo().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
});

ArtifactTypeDefinition tarArtifactTypeDefinition = project.getDependencies().getArtifactTypes().maybeCreate("tar.gz");
project.getDependencies().registerTransform(SymbolicLinkPreservingUntarTransform.class, transformSpec -> {
transformSpec.getFrom().attribute(ArtifactAttributes.ARTIFACT_FORMAT, tarArtifactTypeDefinition.getName());
transformSpec.getTo().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
transformSpec.getFrom().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, tarArtifactTypeDefinition.getName());
transformSpec.getTo().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
});

setupResolutionsContainer(project);
Expand All @@ -80,7 +79,8 @@ private void setupDistributionContainer(Project project, Property<Boolean> docke
distributionsContainer = project.container(ElasticsearchDistribution.class, name -> {
Configuration fileConfiguration = project.getConfigurations().create("es_distro_file_" + name);
Configuration extractedConfiguration = project.getConfigurations().create(DISTRO_EXTRACTED_CONFIG_PREFIX + name);
extractedConfiguration.getAttributes().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.DIRECTORY_TYPE);
extractedConfiguration.getAttributes()
.attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.DIRECTORY_TYPE);
return new ElasticsearchDistribution(
name,
project.getObjects(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@
import org.gradle.api.artifacts.Configuration;
import org.gradle.api.artifacts.type.ArtifactTypeDefinition;
import org.gradle.api.file.RegularFile;
import org.gradle.api.internal.artifacts.ArtifactAttributes;
import org.gradle.api.plugins.BasePlugin;
import org.gradle.api.plugins.JavaPlugin;
import org.gradle.api.plugins.JavaPluginExtension;
Expand Down Expand Up @@ -236,7 +235,7 @@ public Object doCall() {

// also make the zip available as a configuration (used when depending on this project)
Configuration configuration = project.getConfigurations().create("zip");
configuration.getAttributes().attribute(ArtifactAttributes.ARTIFACT_FORMAT, ArtifactTypeDefinition.ZIP_TYPE);
configuration.getAttributes().attribute(ArtifactTypeDefinition.ARTIFACT_TYPE_ATTRIBUTE, ArtifactTypeDefinition.ZIP_TYPE);
project.getArtifacts().add("zip", bundle);

return bundle;
Expand Down
Loading

0 comments on commit 7367932

Please sign in to comment.