Skip to content

Commit

Permalink
Merge branch 'main' into wrr-state-across-requests
Browse files Browse the repository at this point in the history
  • Loading branch information
Anshu Agarwal committed Jan 25, 2023
2 parents 2007e8b + 715ff72 commit b1ca890
Show file tree
Hide file tree
Showing 101 changed files with 824 additions and 271 deletions.
1 change: 1 addition & 0 deletions .ci/bwcVersions
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ BWC_VERSION:
- "2.4.1"
- "2.4.2"
- "2.5.0"
- "2.5.1"
- "2.6.0"
9 changes: 9 additions & 0 deletions .github/ISSUE_TEMPLATE/failed_check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
title: Gradle Check Failure.
labels: >test-failure bug
---

A gradle check workflow has failed after merge.

PR: {{ env.workflow_url }}
CommitId: {{ env.pr_from_sha }}
14 changes: 12 additions & 2 deletions .github/workflows/backport.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,22 @@ on:

jobs:
backport:
if: ${{ contains(github.event.label.name, 'backport') }}
name: Backport
runs-on: ubuntu-latest
# Only react to merged PRs for security reasons.
# See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target.
if: >
github.event.pull_request.merged
&& (
github.event.action == 'closed'
|| (
github.event.action == 'labeled'
&& contains(github.event.label.name, 'backport')
)
)
permissions:
contents: write
pull-requests: write
name: Backport
steps:
- name: GitHub App token
id: github_app_token
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dependabot_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
- name: Update the changelog
uses: dangoslen/dependabot-changelog-helper@v1
with:
version: 'Unreleased'
version: 'Unreleased 3.0'

- name: Commit the changes
uses: stefanzweifel/git-auto-commit-action@v4
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/gradle-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
permissions:
contents: read # to fetch code (actions/checkout)
pull-requests: write # to create or update comment (peter-evans/create-or-update-comment)
issues: write # To create an issue if check fails on push.

runs-on: ubuntu-latest
timeout-minutes: 130
Expand Down Expand Up @@ -123,3 +124,12 @@ jobs:
* **CommitID:** ${{ env.pr_from_sha }}
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?
- name: Create Issue On Push Failure
if: ${{ github.event_name == 'push' && failure() }}
uses: dblock/create-a-github-issue@v3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
assignees: ${{ github.event.head_commit.author.username }}, ${{ github.triggering_actor }}
filename: .github/ISSUE_TEMPLATE/failed_check.md
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@ out/
!.idea/vcs.xml
!.idea/icon.svg

# These files are generated in the main tree by IntelliJ
# These files are generated in the main tree by annotation processors
benchmarks/src/main/generated/*
benchmarks/bin/*
benchmarks/build-eclipse-default/*

# eclipse files
.project
Expand Down
59 changes: 17 additions & 42 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,27 @@

# CHANGELOG
## [Unreleased]
### Dependencies
- Bumps `azure-core-http-netty` from 1.12.7 to 1.12.8
- Bumps `reactor-netty` from 1.1.1 to 1.1.2


All notable changes to this project are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See the [CONTRIBUTING guide](./CONTRIBUTING.md#Changelog) for instructions on how to add changelog entries.

## [Unreleased 3.0]
### Added
- Add support of default replica count cluster setting ([#5610](https://github.com/opensearch-project/OpenSearch/pull/5610))
- Hardened token permissions in GitHub workflows ([#4587](https://github.com/opensearch-project/OpenSearch/pull/4587))
- Support for HTTP/2 (server-side) ([#3847](https://github.com/opensearch-project/OpenSearch/pull/3847))
- Add getter for path field in NestedQueryBuilder ([#4636](https://github.com/opensearch-project/OpenSearch/pull/4636))
- Apply reproducible builds configuration for OpenSearch plugins through gradle plugin ([#4746](https://github.com/opensearch-project/OpenSearch/pull/4746))
- Add project health badges to the README.md ([#4843](https://github.com/opensearch-project/OpenSearch/pull/4843))
- [Test] Add IAE test for deprecated edgeNGram analyzer name ([#5040](https://github.com/opensearch-project/OpenSearch/pull/5040))
- Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4+ ([#5151](https://github.com/opensearch-project/OpenSearch/pull/5151))
- Add feature flag for extensions ([#5211](https://github.com/opensearch-project/OpenSearch/pull/5211))
- Support to fail open requests on search shard failures with weighted traffic routing ([#5072](https://github.com/opensearch-project/OpenSearch/pull/5072))
- Added jackson dependency to server ([#5366] (https://github.com/opensearch-project/OpenSearch/pull/5366))
- Adding support to register settings dynamically ([#5495](https://github.com/opensearch-project/OpenSearch/pull/5495))
- Added experimental support for extensions ([#5347](https://github.com/opensearch-project/OpenSearch/pull/5347)), ([#5518](https://github.com/opensearch-project/OpenSearch/pull/5518), ([#5597](https://github.com/opensearch-project/OpenSearch/pull/5597)), ([#5615](https://github.com/opensearch-project/OpenSearch/pull/5615)))
- Add CI bundle pattern to distribution download ([#5348](https://github.com/opensearch-project/OpenSearch/pull/5348))
- Add support for ppc64le architecture ([#5459](https://github.com/opensearch-project/OpenSearch/pull/5459))
- Support versioning for Weighted routing apis([#5255](https://github.com/opensearch-project/OpenSearch/pull/5255))
- Added @gbbafna as an OpenSearch maintainer ([#5668](https://github.com/opensearch-project/OpenSearch/pull/5668))
- Add support for discovered cluster manager and remove local weights ([#5680](https://github.com/opensearch-project/OpenSearch/pull/5680))
- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))
- Add query for initialized extensions ([#5658](https://github.com/opensearch-project/OpenSearch/pull/5658))
- Revert 'Added jackson dependency to server' and change extension reading ([#5768](https://github.com/opensearch-project/OpenSearch/pull/5768))
- Add support to disallow search request with preference parameter with strict weighted shard routing([#5874](https://github.com/opensearch-project/OpenSearch/pull/5874))
- Replace latches with CompletableFutures for extensions ([#5646](https://github.com/opensearch-project/OpenSearch/pull/5646))
### Dependencies
- Bumps `wiremock-jre8-standalone` from 2.33.2 to 2.35.0
- Bumps `gson` from 2.10 to 2.10.1
- Bumps `json-schema-validator` from 1.0.73 to 1.0.76

### Dependencies
- Bumps `log4j-core` from 2.18.0 to 2.19.0
- Bumps `reactor-netty-http` from 1.0.18 to 1.0.23
- Bumps `jettison` from 1.5.0 to 1.5.3
- Bumps `forbiddenapis` from 3.3 to 3.4
- Bumps `gson` from 2.9.0 to 2.10
- Bumps `avro` from 1.11.0 to 1.11.1
- Bumps `woodstox-core` from 6.3.0 to 6.3.1
- Bumps `xmlbeans` from 5.1.0 to 5.1.1 ([#4354](https://github.com/opensearch-project/OpenSearch/pull/4354))
Expand All @@ -51,32 +32,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bumps `com.diffplug.spotless` from 6.10.0 to 6.11.0 ([#4547](https://github.com/opensearch-project/OpenSearch/pull/4547))
- Bumps `reactor-core` from 3.4.23 to 3.5.1 ([#5604](https://github.com/opensearch-project/OpenSearch/pull/5604))
- Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
- Bumps `commons-compress` from 1.21 to 1.22
- Bumps `jcodings` from 1.0.57 to 1.0.58 ([#5233](https://github.com/opensearch-project/OpenSearch/pull/5233))
- Bumps `google-http-client-jackson2` from 1.35.0 to 1.42.3 ([#5234](https://github.com/opensearch-project/OpenSearch/pull/5234))
- Bumps `azure-core` from 1.33.0 to 1.34.0 ([#5235](https://github.com/opensearch-project/OpenSearch/pull/5235))
- Bumps `azure-core-http-netty` from 1.12.4 to 1.12.7 ([#5235](https://github.com/opensearch-project/OpenSearch/pull/5235))
- Bumps `spock-core` from 2.1-groovy-3.0 to 2.3-groovy-3.0 ([#5315](https://github.com/opensearch-project/OpenSearch/pull/5315))
- Bumps `json-schema-validator` from 1.0.69 to 1.0.73 ([#5316](https://github.com/opensearch-project/OpenSearch/pull/5316))
- Bumps `proto-google-common-protos` from 2.8.0 to 2.10.0 ([#5318](https://github.com/opensearch-project/OpenSearch/pull/5318))
- Update to Gradle 7.6 and JDK-19 ([#4973](https://github.com/opensearch-project/OpenSearch/pull/4973))
- Update Apache Lucene to 9.5.0-snapshot-d5cef1c ([#5570](https://github.com/opensearch-project/OpenSearch/pull/5570))
- Bump antlr4 from 4.9.3 to 4.11.1 ([#4546](https://github.com/opensearch-project/OpenSearch/pull/4546))
- Bumps `maven-model` from 3.6.2 to 3.8.6 ([#5599](https://github.com/opensearch-project/OpenSearch/pull/5599))
- Bumps `maxmind-db` from 2.1.0 to 3.0.0 ([#5601](https://github.com/opensearch-project/OpenSearch/pull/5601))
- Bumps `protobuf-java` from 3.21.11 to 3.21.12 ([#5603](https://github.com/opensearch-project/OpenSearch/pull/5603))
- Update nebula-publishing-plugin to 19.2.0 ([#5704](https://github.com/opensearch-project/OpenSearch/pull/5704))
- Bumps `azure-core-http-netty` from 1.12.7 to 1.12.8

### Changed
- [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))
- Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
- Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
- Migrate client transports to Apache HttpClient / Core 5.x ([#4459](https://github.com/opensearch-project/OpenSearch/pull/4459))
- Support remote translog transfer for request level durability([#4480](https://github.com/opensearch-project/OpenSearch/pull/4480))
- Changed http code on create index API with bad input raising NotXContentException from 500 to 400 ([#4773](https://github.com/opensearch-project/OpenSearch/pull/4773))
- Change http code for DecommissioningFailedException from 500 to 400 ([#5283](https://github.com/opensearch-project/OpenSearch/pull/5283))
- Pre conditions check before updating weighted routing metadata ([#4955](https://github.com/opensearch-project/OpenSearch/pull/4955))
- Gracefully handle concurrent zone decommission action ([#5542](https://github.com/opensearch-project/OpenSearch/pull/5542))

### Deprecated

Expand All @@ -98,29 +68,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Fixed
- Fix 'org.apache.hc.core5.http.ParseException: Invalid protocol version' under JDK 16+ ([#4827](https://github.com/opensearch-project/OpenSearch/pull/4827))
- Fixed compression support for h2c protocol ([#4944](https://github.com/opensearch-project/OpenSearch/pull/4944))
- Reject bulk requests with invalid actions ([#5299](https://github.com/opensearch-project/OpenSearch/issues/5299))
- Support OpenSSL Provider with default Netty allocator ([#5460](https://github.com/opensearch-project/OpenSearch/pull/5460))
- Increasing timeout of testQuorumRecovery to 90 seconds from 30 ([#5651](https://github.com/opensearch-project/OpenSearch/pull/5651))
- [Segment Replication] Fix for peer recovery ([#5344](https://github.com/opensearch-project/OpenSearch/pull/5344))
- [Test] Renaming PIT tests to IT to fix intermittent test failures ([#5750](https://github.com/opensearch-project/OpenSearch/pull/5750))

### Security

## [Unreleased 2.x]
### Added
- Adding index create block when all nodes have breached high disk watermark ([#5852](https://github.com/opensearch-project/OpenSearch/pull/5852))
- Added cluster manager throttling stats in nodes/stats API ([#5790](https://github.com/opensearch-project/OpenSearch/pull/5790))
- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))
- Add query for initialized extensions ([#5658](https://github.com/opensearch-project/OpenSearch/pull/5658))
- Add update-index-settings allowlist for searchable snapshot ([#5907](https://github.com/opensearch-project/OpenSearch/pull/5907))

### Dependencies
- Update nebula-publishing-plugin to 19.2.0 ([#5704](https://github.com/opensearch-project/OpenSearch/pull/5704))
- Bumps `reactor-netty` from 1.1.1 to 1.1.2

### Changed
- Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
- [Refactor] Use local opensearch.common.SetOnce instead of lucene's utility class ([#5947](https://github.com/opensearch-project/OpenSearch/pull/5947))

### Deprecated

### Removed

### Fixed
- [Segment Replication] Fix for peer recovery ([#5344](https://github.com/opensearch-project/OpenSearch/pull/5344))

### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.4...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.4...2.x
[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
4 changes: 2 additions & 2 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -118,14 +118,14 @@ dependencies {
api 'com.avast.gradle:gradle-docker-compose-plugin:0.15.2'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.8.6'
api 'com.networknt:json-schema-validator:1.0.73'
api 'com.networknt:json-schema-validator:1.0.76'
api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"

testFixturesApi "junit:junit:${props.getProperty('junit')}"
testFixturesApi "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
testFixturesApi gradleApi()
testFixturesApi gradleTestKit()
testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone:2.33.2'
testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone:2.35.0'
testImplementation "org.mockito:mockito-core:${props.getProperty('mockito')}"
integTestImplementation('org.spockframework:spock-core:2.3-groovy-3.0') {
exclude module: "groovy"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ java.nio.channels.SocketChannel#connect(java.net.SocketAddress)
java.lang.Boolean#getBoolean(java.lang.String)

org.apache.lucene.util.IOUtils @ use @org.opensearch.core.internal.io instead
org.apache.lucene.util.SetOnce @ use @org.opensearch.common.SetOnce instead

@defaultMessage use executors from org.opensearch.common.util.concurrent.OpenSearchExecutors instead which will properly bubble up Errors
java.util.concurrent.AbstractExecutorService#<init>()
Expand Down
104 changes: 104 additions & 0 deletions libs/core/src/main/java/org/opensearch/common/SetOnce.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

/*
* Modifications Copyright OpenSearch Contributors. See
* GitHub history for details.
*/
package org.opensearch.common;

import java.util.concurrent.atomic.AtomicReference;

/**
* A convenient class which offers a semi-immutable object wrapper implementation which allows one
* to set the value of an object exactly once, and retrieve it many times. If {@link #set(Object)}
* is called more than once, {@link AlreadySetException} is thrown and the operation will fail.
*
* This is borrowed from lucene's experimental API. It is not reused to eliminate the dependency
* on lucene core for such a simple (standalone) utility class that may change beyond OpenSearch needs.
*
* @opensearch.api
*/
public final class SetOnce<T> implements Cloneable {

/** Thrown when {@link SetOnce#set(Object)} is called more than once. */
public static final class AlreadySetException extends IllegalStateException {
public AlreadySetException() {
super("The object cannot be set twice!");
}
}

/** Holding object and marking that it was already set */
private static final class Wrapper<T> {
private T object;

private Wrapper(T object) {
this.object = object;
}
}

private final AtomicReference<Wrapper<T>> set;

/**
* A default constructor which does not set the internal object, and allows setting it by calling
* {@link #set(Object)}.
*/
public SetOnce() {
set = new AtomicReference<>();
}

/**
* Creates a new instance with the internal object set to the given object. Note that any calls to
* {@link #set(Object)} afterwards will result in {@link AlreadySetException}
*
* @throws AlreadySetException if called more than once
* @see #set(Object)
*/
public SetOnce(T obj) {
set = new AtomicReference<>(new Wrapper<>(obj));
}

/** Sets the given object. If the object has already been set, an exception is thrown. */
public final void set(T obj) {
if (!trySet(obj)) {
throw new AlreadySetException();
}
}

/**
* Sets the given object if none was set before.
*
* @return true if object was set successfully, false otherwise
*/
public final boolean trySet(T obj) {
return set.compareAndSet(null, new Wrapper<>(obj));
}

/** Returns the object set by {@link #set(Object)}. */
public final T get() {
Wrapper<T> wrapper = set.get();
return wrapper == null ? null : wrapper.object;
}
}
Loading

0 comments on commit b1ca890

Please sign in to comment.