Skip to content

Commit

Permalink
Merge branch 'master' into pr/23767
Browse files Browse the repository at this point in the history 
* master: (356 commits)
  Do not set SO_LINGER on server channels (elastic#26997)
  Fix inconsistencies in the rest api specs for *_script (elastic#26971)
  fix inconsistencies in the rest api specs for cat.snapshots (elastic#26996)
  Add docs on full_id parameter in cat nodes API
  [TEST] Add test that replicates versioned updates with random flushes
  Use internal searcher for all indexing related operations in the engine
  Reformat paragraph in template docs to 80 columns
  Clarify settings and template on create index
  Fix reference to TcpTransport in documentation
  Allow Uid#decodeId to decode from a byte array slice (elastic#26987)
  Fix a typo in the similarity docs (elastic#26970)
  Use separate searchers for "search visibility" vs "move indexing buffer to disk (elastic#26972)
  Create weights lazily in filter and filters aggregation (elastic#26983)
  Use a dedicated ThreadGroup in rest sniffer (elastic#26897)
  Fire global checkpoint sync under system context
  Update by Query is modified to accept short `script` parameter. (elastic#26841)
  Cat shards bytes (elastic#26952)
  Add support for parsing inline script (elastic#23824) (elastic#26846)
  Change default value to true for transpositions parameter of fuzzy query (elastic#26901)
  Adding unreleased 5.6.4 version number to Version.java
  ...
  • Loading branch information
@jasontedor
jasontedor committed Oct 16, 2017
2 parents c6ad0cc + 277637f commit 6bb1d2c
Show file tree
Hide file tree
Showing 1,389 changed files with 27,794 additions and 16,291 deletions.
18 changes: 15 additions & 3 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ If you have a bugfix or new feature that you would like to contribute to Elastic

We enjoy working with contributors to get their code accepted. There are many approaches to fixing a problem and it is important to find the best approach before writing too much code.

Note that it is unlikely the project will merge refactors for the sake of refactoring. These
types of pull requests have a high cost to maintainers in reviewing and testing with little
to no tangible benefit. This especially includes changes generated by tools. For example,
converting all generic interface instances to use the diamond operator.

The process for contributing to any of the [Elastic repositories](https://github.com/elastic/) is similar. Details for individual projects can be found below.

### Fork and clone the repository
Expand Down Expand Up @@ -106,9 +111,16 @@ then `File->New Project From Existing Sources`. Point to the root of
the source directory, select
`Import project from external model->Gradle`, enable
`Use auto-import`. Additionally, in order to run tests directly from
IDEA 2017.2 and above it is required to disable IDEA run launcher,
which can be achieved by adding `-Didea.no.launcher=true`
[JVM option](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties)
IDEA 2017.2 and above it is required to disable IDEA run launcher to avoid
finding yourself in "jar hell", which can be achieved by adding the
`-Didea.no.launcher=true` [JVM
option](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties)
or by adding `idea.no.launcher=true` to the
`idea.properties`[https://www.jetbrains.com/help/idea/file-idea-properties.html]
file which can be accessed under Help > Edit Custom Properties within IDEA. You
may also need to [remove `ant-javafx.jar` from your
classpath][https://github.com/elastic/elasticsearch/issues/14348] if that is
reported as a source of jar hell.

The Elasticsearch codebase makes heavy use of Java `assert`s and the
test runner requires that assertions be enabled within the JVM. This
Expand Down
25 changes: 14 additions & 11 deletions TESTING.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,7 @@ These are the linux flavors the Vagrantfile currently supports:
* centos-6
* centos-7
* fedora-25
* fedora-26
* oel-6 aka Oracle Enterprise Linux 6
* oel-7 aka Oracle Enterprise Linux 7
* sles-12
Expand Down Expand Up @@ -471,28 +472,30 @@ is tested depends on the branch. On master, this will test against the current
stable branch. On the stable branch, it will test against the latest release
branch. Finally, on a release branch, it will test against the most recent release.

=== BWC Testing against a specific branch
=== BWC Testing against a specific remote/branch

Sometimes a backward compatibility change spans two versions. A common case is a new functionality
that needs a BWC bridge in and an unreleased versioned of a release branch (for example, 5.x).
To test the changes, you can instruct gradle to build the BWC version from a local branch instead of
pulling the release branch from GitHub. You do so using the `tests.bwc.refspec` system property:
To test the changes, you can instruct gradle to build the BWC version from a another remote/branch combination instead of
pulling the release branch from GitHub. You do so using the `tests.bwc.remote` and `tests.bwc.refspec` system properties:

-------------------------------------------------
gradle check -Dtests.bwc.refspec=origin/index_req_bwc_5.x
gradle check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x
-------------------------------------------------

The branch needs to be available on the local clone that the BWC makes of the repository you run the
tests from. Using the `origin` remote is a handy trick to make sure that a branch is available
and is up to date in the case of multiple runs.
The branch needs to be available on the remote that the BWC makes of the
repository you run the tests from. Using the remote is a handy trick to make
sure that a branch is available and is up to date in the case of multiple runs.

Example:

Say you need to make a change to `master` and have a BWC layer in `5.x`. You will need to:
. Create a branch called `index_req_change` off `master`. This will contain your change.
Say you need to make a change to `master` and have a BWC layer in `5.x`. You
will need to:
. Create a branch called `index_req_change` off your remote `${remote}`. This
will contain your change.
. Create a branch called `index_req_bwc_5.x` off `5.x`. This will contain your bwc layer.
. If not running the tests locally, push both branches to your remote repository.
. Run the tests with `gradle check -Dtests.bwc.refspec=origin/index_req_bwc_5.x`
. Push both branches to your remote repository.
. Run the tests with `gradle check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x`.

== Coverage analysis

Expand Down
4 changes: 4 additions & 0 deletions Vagrantfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ Vagrant.configure(2) do |config|
config.vm.box = "elastic/fedora-25-x86_64"
dnf_common config
end
config.vm.define "fedora-26" do |config|
config.vm.box = "elastic/fedora-26-x86_64"
dnf_common config
end
config.vm.define "opensuse-42" do |config|
config.vm.box = "elastic/opensuse-42-x86_64"
opensuse_common config
Expand Down
75 changes: 47 additions & 28 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import org.eclipse.jgit.lib.Repository
import org.eclipse.jgit.lib.RepositoryBuilder
import org.gradle.plugins.ide.eclipse.model.SourceFolder
import org.apache.tools.ant.taskdefs.condition.Os
import org.elasticsearch.gradle.BuildPlugin
import org.elasticsearch.gradle.VersionProperties
import org.elasticsearch.gradle.Version

Expand Down Expand Up @@ -60,6 +61,10 @@ configure(subprojects.findAll { it.projectDir.toPath().startsWith(rootPath) }) {
}
}
}
plugins.withType(BuildPlugin).whenPluginAdded {
project.licenseFile = project.rootProject.file('LICENSE.txt')
project.noticeFile = project.rootProject.file('NOTICE.txt')
}
}

/* Introspect all versions of ES that may be tested agains for backwards
Expand Down Expand Up @@ -122,6 +127,16 @@ if (currentVersion.bugfix == 0) {
}
}

// build metadata from previous build, contains eg hashes for bwc builds
String buildMetadataValue = System.getenv('BUILD_METADATA')
if (buildMetadataValue == null) {
buildMetadataValue = ''
}
Map<String, String> buildMetadataMap = buildMetadataValue.tokenize(';').collectEntries {
def (String key, String value) = it.split('=')
return [key, value]
}

// injecting groovy property variables into all projects
allprojects {
project.ext {
Expand All @@ -131,6 +146,7 @@ allprojects {
// for backcompat testing
indexCompatVersions = versions
wireCompatVersions = versions.subList(prevMinorIndex, versions.size())
buildMetadata = buildMetadataMap
}
}

Expand Down Expand Up @@ -188,33 +204,15 @@ task branchConsistency {
}

subprojects {
project.afterEvaluate {
// include license and notice in jars
tasks.withType(Jar) {
into('META-INF') {
from project.rootProject.rootDir
include 'LICENSE.txt'
include 'NOTICE.txt'
}
}
// ignore missing javadocs
tasks.withType(Javadoc) { Javadoc javadoc ->
// the -quiet here is because of a bug in gradle, in that adding a string option
// by itself is not added to the options. By adding quiet, both this option and
// the "value" -quiet is added, separated by a space. This is ok since the javadoc
// command already adds -quiet, so we are just duplicating it
// see https://discuss.gradle.org/t/add-custom-javadoc-option-that-does-not-take-an-argument/5959
javadoc.options.encoding='UTF8'
javadoc.options.addStringOption('Xdoclint:all,-missing', '-quiet')
/*
TODO: building javadocs with java 9 b118 is currently broken with weird errors, so
for now this is commented out...try again with the next ea build...
javadoc.executable = new File(project.javaHome, 'bin/javadoc')
if (project.javaVersion == JavaVersion.VERSION_1_9) {
// TODO: remove this hack! gradle should be passing this...
javadoc.options.addStringOption('source', '8')
}*/
}
// ignore missing javadocs
tasks.withType(Javadoc) { Javadoc javadoc ->
// the -quiet here is because of a bug in gradle, in that adding a string option
// by itself is not added to the options. By adding quiet, both this option and
// the "value" -quiet is added, separated by a space. This is ok since the javadoc
// command already adds -quiet, so we are just duplicating it
// see https://discuss.gradle.org/t/add-custom-javadoc-option-that-does-not-take-an-argument/5959
javadoc.options.encoding='UTF8'
javadoc.options.addStringOption('Xdoclint:all,-missing', '-quiet')
}

/* Sets up the dependencies that we build as part of this project but
Expand Down Expand Up @@ -272,6 +270,27 @@ subprojects {
}
}
}

// Handle javadoc dependencies across projects. Order matters: the linksOffline for
// org.elasticsearch:elasticsearch must be the last one or all the links for the
// other packages (e.g org.elasticsearch.client) will point to core rather than
// their own artifacts.
if (project.plugins.hasPlugin(BuildPlugin)) {
String artifactsHost = VersionProperties.elasticsearch.endsWith("-SNAPSHOT") ? "https://snapshots.elastic.co" : "https://artifacts.elastic.co"
Closure sortClosure = { a, b -> b.group <=> a.group }
Closure depJavadocClosure = { dep ->
if (dep.group != null && dep.group.startsWith('org.elasticsearch')) {
String substitution = project.ext.projectSubstitutions.get("${dep.group}:${dep.name}:${dep.version}")
if (substitution != null) {
project.javadoc.dependsOn substitution + ':javadoc'
String artifactPath = dep.group.replaceAll('\\.', '/') + '/' + dep.name.replaceAll('\\.', '/') + '/' + dep.version
project.javadoc.options.linksOffline artifactsHost + "/javadoc/" + artifactPath, "${project.project(substitution).buildDir}/docs/javadoc/"
}
}
}
project.configurations.compile.dependencies.findAll().toSorted(sortClosure).each(depJavadocClosure)
project.configurations.provided.dependencies.findAll().toSorted(sortClosure).each(depJavadocClosure)
}
}
}

Expand All @@ -281,7 +300,7 @@ subprojects {
// the dependency is added.
gradle.projectsEvaluated {
allprojects {
if (project.path == ':test:framework' || project.path == ':client:test') {
if (project.path == ':test:framework') {
// :test:framework:test cannot run before and after :core:test
return
}
Expand Down
3 changes: 2 additions & 1 deletion buildSrc/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,9 @@ dependencies {
compile 'com.netflix.nebula:gradle-info-plugin:3.0.3'
compile 'org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'
compile 'com.perforce:p4java:2012.3.551082' // THIS IS SUPPOSED TO BE OPTIONAL IN THE FUTURE....
compile 'de.thetaphi:forbiddenapis:2.3'
compile 'de.thetaphi:forbiddenapis:2.4.1'
compile 'org.apache.rat:apache-rat:0.11'
compile "org.elasticsearch:jna:4.4.0-1"
}

// Gradle 2.14+ removed ProgressLogger(-Factory) classes from the public APIs
Expand Down
46 changes: 21 additions & 25 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ import org.gradle.api.Project
import org.gradle.api.Task
import org.gradle.api.XmlProvider
import org.gradle.api.artifacts.Configuration
import org.gradle.api.artifacts.Dependency
import org.gradle.api.artifacts.ModuleDependency
import org.gradle.api.artifacts.ModuleVersionIdentifier
import org.gradle.api.artifacts.ProjectDependency
import org.gradle.api.artifacts.ResolvedArtifact
import org.gradle.api.artifacts.dsl.RepositoryHandler
import org.gradle.api.file.CopySpec
import org.gradle.api.plugins.JavaPlugin
import org.gradle.api.publish.maven.MavenPublication
import org.gradle.api.publish.maven.plugins.MavenPublishPlugin
Expand Down Expand Up @@ -270,8 +270,8 @@ class BuildPlugin implements Plugin<Project> {
})

// force all dependencies added directly to compile/testCompile to be non-transitive, except for ES itself
Closure disableTransitiveDeps = { Dependency dep ->
if (dep instanceof ModuleDependency && !(dep instanceof ProjectDependency) && dep.group.startsWith('org.elasticsearch') == false) {
Closure disableTransitiveDeps = { ModuleDependency dep ->
if (!(dep instanceof ProjectDependency) && dep.group.startsWith('org.elasticsearch') == false) {
dep.transitive = false

// also create a configuration just for this dependency version, so that later
Expand Down Expand Up @@ -455,28 +455,8 @@ class BuildPlugin implements Plugin<Project> {
}

static void configureJavadoc(Project project) {
String artifactsHost = VersionProperties.elasticsearch.endsWith("-SNAPSHOT") ? "https://snapshots.elastic.co" : "https://artifacts.elastic.co"
project.afterEvaluate {
project.tasks.withType(Javadoc) {
executable = new File(project.javaHome, 'bin/javadoc')
}
/*
* Order matters, the linksOffline for org.elasticsearch:elasticsearch must be the last one
* or all the links for the other packages (e.g org.elasticsearch.client) will point to core rather than their own artifacts
*/
Closure sortClosure = { a, b -> b.group <=> a.group }
Closure depJavadocClosure = { dep ->
if (dep.group != null && dep.group.startsWith('org.elasticsearch')) {
String substitution = project.ext.projectSubstitutions.get("${dep.group}:${dep.name}:${dep.version}")
if (substitution != null) {
project.javadoc.dependsOn substitution + ':javadoc'
String artifactPath = dep.group.replaceAll('\\.', '/') + '/' + dep.name.replaceAll('\\.', '/') + '/' + dep.version
project.javadoc.options.linksOffline artifactsHost + "/javadoc/" + artifactPath, "${project.project(substitution).buildDir}/docs/javadoc/"
}
}
}
project.configurations.compile.dependencies.findAll().toSorted(sortClosure).each(depJavadocClosure)
project.configurations.provided.dependencies.findAll().toSorted(sortClosure).each(depJavadocClosure)
project.tasks.withType(Javadoc) {
executable = new File(project.javaHome, 'bin/javadoc')
}
configureJavadocJar(project)
}
Expand All @@ -502,6 +482,8 @@ class BuildPlugin implements Plugin<Project> {

/** Adds additional manifest info to jars */
static void configureJars(Project project) {
project.ext.licenseFile = null
project.ext.noticeFile = null
project.tasks.withType(Jar) { Jar jarTask ->
// we put all our distributable files under distributions
jarTask.destinationDir = new File(project.buildDir, 'distributions')
Expand All @@ -525,6 +507,20 @@ class BuildPlugin implements Plugin<Project> {
jarTask.manifest.attributes('Change': 'Unknown')
}
}
// add license/notice files
project.afterEvaluate {
if (project.licenseFile == null || project.noticeFile == null) {
throw new GradleException("Must specify license and notice file for project ${project.path}")
}
jarTask.into('META-INF') {
from(project.licenseFile.parent) {
include project.licenseFile.name
}
from(project.noticeFile.parent) {
include project.noticeFile.name
}
}
}
}
}

Expand Down
2 changes: 1 addition & 1 deletion buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ public class RestTestsFromSnippetsTask extends SnippetsTask {
*
* `sh` snippets that contain `curl` almost always should be marked
* with `// CONSOLE`. In the exceptionally rare cases where they are
* not communicating with Elasticsearch, like the xamples in the ec2
* not communicating with Elasticsearch, like the examples in the ec2
* and gce discovery plugins, the snippets should be marked
* `// NOTCONSOLE`. */
return snippet.language == 'js' || snippet.curl
Expand Down
4 changes: 4 additions & 0 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesExtension.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ class PluginPropertiesExtension {
@Input
boolean hasClientJar = false

/** True if the plugin requires the elasticsearch keystore to exist, false otherwise. */
@Input
boolean requiresKeystore = false

/** A license file that should be included in the built plugin zip. */
@Input
File licenseFile = null
Expand Down
3 changes: 2 additions & 1 deletion buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesTask.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ class PluginPropertiesTask extends Copy {
'elasticsearchVersion': stringSnap(VersionProperties.elasticsearch),
'javaVersion': project.targetCompatibility as String,
'classname': extension.classname,
'hasNativeController': extension.hasNativeController
'hasNativeController': extension.hasNativeController,
'requiresKeystore': extension.requiresKeystore
]
}
}
5 changes: 0 additions & 5 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/precommit/ForbiddenPatternsTask.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,6 @@ public class ForbiddenPatternsTask extends DefaultTask {
patterns.put('nocommit should be all lowercase or all uppercase',
/((?i)nocommit)(?<!(nocommit|NOCOMMIT))/)
patterns.put('tab', /\t/)
if (System.getProperty('build.snapshot', 'true').equals('false')) {
patterns.put('norelease', /norelease|NORELEASE/)
}
patterns.put('norelease should be all lowercase or all uppercase',
/((?i)norelease)(?<!(norelease|NORELEASE))/)


inputs.property("excludes", filesFilter.excludes)
Expand Down
7 changes: 1 addition & 6 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/precommit/ThirdPartyAuditTask.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ import org.apache.tools.ant.DefaultLogger;
import org.apache.tools.ant.Project;
import org.elasticsearch.gradle.AntTask;
import org.gradle.api.artifacts.Configuration;
import org.gradle.api.artifacts.FileCollectionDependency;
import org.gradle.api.file.FileCollection;
import org.gradle.api.tasks.Input
import org.gradle.api.tasks.InputFiles
Expand Down Expand Up @@ -86,11 +85,7 @@ public class ThirdPartyAuditTask extends AntTask {

// we only want third party dependencies.
jars = configuration.fileCollection({ dependency ->
// include SelfResolvingDependency with files in the validation
if (dependency instanceof FileCollectionDependency) {
return true
}
return dependency.group && dependency.group.startsWith("org.elasticsearch") == false
dependency.group.startsWith("org.elasticsearch") == false
});

// we don't want provided dependencies, which we have already scanned. e.g. don't
Expand Down
8 changes: 3 additions & 5 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterConfiguration.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,11 @@ class ClusterConfiguration {
boolean debug = false

/**
* if <code>true</code> each node will be configured with <tt>discovery.zen.minimum_master_nodes</tt> set
* to the total number of nodes in the cluster. This will also cause that each node has `0s` state recovery
* timeout which can lead to issues if for instance an existing clusterstate is expected to be recovered
* before any tests start
* Configuration of the setting <tt>discovery.zen.minimum_master_nodes</tt> on the nodes.
* In case of more than one node, this defaults to (number of nodes / 2) + 1
*/
@Input
boolean useMinimumMasterNodes = true
Closure<Integer> minimumMasterNodes = { getNumNodes() > 1 ? getNumNodes().intdiv(2) + 1 : -1 }

@Input
String jvmArgs = "-Xms" + System.getProperty('tests.heap.size', '512m') +
Expand Down
Loading

0 comments on commit 6bb1d2c

Please sign in to comment.