Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/main' into remote-state-stats
Browse files Browse the repository at this point in the history
  • Loading branch information
Aman Khare committed Oct 18, 2023
2 parents 198e2fa + 9cc0e7d commit 0953f54
Show file tree
Hide file tree
Showing 127 changed files with 2,633 additions and 235 deletions.
58 changes: 14 additions & 44 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add events correlation engine plugin ([#6854](https://github.com/opensearch-project/OpenSearch/issues/6854))
- Introduce new dynamic cluster setting to control slice computation for concurrent segment search ([#9107](https://github.com/opensearch-project/OpenSearch/pull/9107))
- Implement on behalf of token passing for extensions ([#8679](https://github.com/opensearch-project/OpenSearch/pull/8679))
- Implement Visitor Design pattern in QueryBuilder to enable the capability to traverse through the complex QueryBuilder tree. ([#10110](https://github.com/opensearch-project/OpenSearch/pull/10110))
- Provide service accounts tokens to extensions ([#9618](https://github.com/opensearch-project/OpenSearch/pull/9618))
- Configurable merge policy for index with an option to choose from LogByteSize and Tiered merge policy ([#9992](https://github.com/opensearch-project/OpenSearch/pull/9992))
- [Admission control] Add enhancements to FS stats to include read/write time, queue size and IO time ([#10541](https://github.com/opensearch-project/OpenSearch/pull/10541))
- [Admission control] Add Resource usage collector service and resource usage tracker ([#9890](https://github.com/opensearch-project/OpenSearch/pull/9890))
- Change file names for remote cluster state ([#10557](https://github.com/opensearch-project/OpenSearch/pull/10557))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand All @@ -33,7 +34,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.google.code.gson:gson` from 2.10 to 2.10.1
- Bump `com.maxmind.geoip2:geoip2` from 4.0.0 to 4.0.1
- Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.16.11 to 0.16.12
- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0
- Bump `org.apache.commons:commons-configuration2` from 2.8.0 to 2.9.0
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.3.0
- Bump `io.opencensus:opencensus-api` from 0.18.0 to 0.31.1 ([#7291](https://github.com/opensearch-project/OpenSearch/pull/7291))
Expand All @@ -46,7 +46,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump JNA version from 5.5 to 5.13 ([#9963](https://github.com/opensearch-project/OpenSearch/pull/9963))
- Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 ([#9822](https://github.com/opensearch-project/OpenSearch/pull/9822))
- Bump `org.eclipse.jgit` from 6.5.0 to 6.7.0 ([#10147](https://github.com/opensearch-project/OpenSearch/pull/10147))
- Bump `netty` from 4.1.99.Final to 4.1.100.Final ([#10564](https://github.com/opensearch-project/OpenSearch/pull/10564))
- Bump OpenTelemetry from 1.30.1 to 1.31.0 ([#10617](https://github.com/opensearch-project/OpenSearch/pull/10617))

### Changed
- [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))
Expand Down Expand Up @@ -96,70 +96,40 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add unreferenced file cleanup count to merge stats ([#10204](https://github.com/opensearch-project/OpenSearch/pull/10204))
- [Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository ([#9839](https://github.com/opensearch-project/OpenSearch/pull/9839))
- Improve compressed request handling ([#10261](https://github.com/opensearch-project/OpenSearch/pull/10261))
- Add cluster state stats
- Per request phase latency ([#10351](https://github.com/opensearch-project/OpenSearch/issues/10351))
- Add cluster state stats ([#10670](https://github.com/opensearch-project/OpenSearch/pull/10670))

### Dependencies
- Bump `peter-evans/create-or-update-comment` from 2 to 3 ([#9575](https://github.com/opensearch-project/OpenSearch/pull/9575))
- Bump `actions/checkout` from 2 to 4 ([#9968](https://github.com/opensearch-project/OpenSearch/pull/9968))
- Bump OpenTelemetry from 1.26.0 to 1.30.1 ([#9950](https://github.com/opensearch-project/OpenSearch/pull/9950))
- Bump `org.apache.commons:commons-compress` from 1.23.0 to 1.24.0 ([#9973, #9972](https://github.com/opensearch-project/OpenSearch/pull/9973, https://github.com/opensearch-project/OpenSearch/pull/9972))
- Bump `com.google.cloud:google-cloud-core-http` from 2.21.1 to 2.23.0 ([#9971](https://github.com/opensearch-project/OpenSearch/pull/9971))
- Bump `mockito` from 5.4.0 to 5.5.0 ([#10022](https://github.com/opensearch-project/OpenSearch/pull/10022))
- Bump `bytebuddy` from 1.14.3 to 1.14.7 ([#10022](https://github.com/opensearch-project/OpenSearch/pull/10022))
- Bump `com.zaxxer:SparseBitSet` from 1.2 to 1.3 ([#10098](https://github.com/opensearch-project/OpenSearch/pull/10098))
- Bump `tibdex/github-app-token` from 1.5.0 to 2.1.0 ([#10125](https://github.com/opensearch-project/OpenSearch/pull/10125))
- Bump `org.wiremock:wiremock-standalone` from 2.35.0 to 3.1.0 ([#9752](https://github.com/opensearch-project/OpenSearch/pull/9752))
- Bump `com.google.http-client:google-http-client-jackson2` from 1.43.2 to 1.43.3 ([#10126](https://github.com/opensearch-project/OpenSearch/pull/10126))
- Bump `org.xerial.snappy:snappy-java` from 1.1.10.3 to 1.1.10.5 ([#10206](https://github.com/opensearch-project/OpenSearch/pull/10206), [#10299](https://github.com/opensearch-project/OpenSearch/pull/10299))
- Bump `com.google.api.grpc:proto-google-common-protos` from 2.10.0 to 2.25.1 ([#10208](https://github.com/opensearch-project/OpenSearch/pull/10208), [#10298](https://github.com/opensearch-project/OpenSearch/pull/10298))
- Bump `codecov/codecov-action` from 2 to 3 ([#10209](https://github.com/opensearch-project/OpenSearch/pull/10209))
- Bump `org.bouncycastle:bcpkix-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
- Bump `org.bouncycastle:bcprov-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
- Bump `org.bouncycastle:bcmail-jdk15to18` from 1.75 to 1.76 ([10219](https://github.com/opensearch-project/OpenSearch/pull/10219))`
- Bump Lucene from 9.7.0 to 9.8.0 ([10276](https://github.com/opensearch-project/OpenSearch/pull/10276))
- Bump `com.netflix.nebula.ospackage-base` from 11.4.0 to 11.5.0 ([#10295](https://github.com/opensearch-project/OpenSearch/pull/10295))
- Bump asm from 9.5 to 9.6 ([#10302](https://github.com/opensearch-project/OpenSearch/pull/10302))
- Bump netty from 4.1.97.Final to 4.1.99.Final ([#10303](https://github.com/opensearch-project/OpenSearch/pull/10303))
- Bump `peter-evans/create-pull-request` from 3 to 5 ([#10301](https://github.com/opensearch-project/OpenSearch/pull/10301))
- Bump `org.apache.avro:avro` from 1.11.2 to 1.11.3 ([#10210](https://github.com/opensearch-project/OpenSearch/pull/10210))
- Bump `org.bouncycastle:bc-fips` from 1.0.2.3 to 1.0.2.4 ([#10297](https://github.com/opensearch-project/OpenSearch/pull/10297))
- Bump `org.apache.zookeeper:zookeeper` from 3.9.0 to 3.9.1 ([#10506](https://github.com/opensearch-project/OpenSearch/pull/10506))
- Bump `de.thetaphi:forbiddenapis` from 3.5.1 to 3.6 ([#10508](https://github.com/opensearch-project/OpenSearch/pull/10508))
- Bump `commons-io:commons-io` from 2.13.0 to 2.14.0 ([#10294](https://github.com/opensearch-project/OpenSearch/pull/10294))
- Bump `org.codehaus.woodstox:stax2-api` from 4.2.1 to 4.2.2 ([#10639](https://github.com/opensearch-project/OpenSearch/pull/10639))
- Bump `com.google.http-client:google-http-client` from 1.43.2 to 1.43.3 ([#10635](https://github.com/opensearch-project/OpenSearch/pull/10635))
- Bump `com.squareup.okio:okio` from 3.5.0 to 3.6.0 ([#10637](https://github.com/opensearch-project/OpenSearch/pull/10637))

### Changed
- Add instrumentation in rest and network layer. ([#9415](https://github.com/opensearch-project/OpenSearch/pull/9415))
- Allow parameterization of tests with OpenSearchIntegTestCase.SuiteScopeTestCase annotation ([#9916](https://github.com/opensearch-project/OpenSearch/pull/9916))
- Mute the query profile IT with concurrent execution ([#9840](https://github.com/opensearch-project/OpenSearch/pull/9840))
- Force merge with `only_expunge_deletes` honors max segment size ([#10036](https://github.com/opensearch-project/OpenSearch/pull/10036))
- Add instrumentation in transport service. ([#10042](https://github.com/opensearch-project/OpenSearch/pull/10042))
- [Tracing Framework] Add support for SpanKind. ([#10122](https://github.com/opensearch-project/OpenSearch/pull/10122))
- Pass parent filter to inner query in nested query ([#10246](https://github.com/opensearch-project/OpenSearch/pull/10246))
- Disable concurrent segment search when terminate_after is used ([#10200](https://github.com/opensearch-project/OpenSearch/pull/10200))
- Add instrumentation in Inbound Handler. ([#100143](https://github.com/opensearch-project/OpenSearch/pull/10143))
- Enable remote segment upload backpressure by default ([#10356](https://github.com/opensearch-project/OpenSearch/pull/10356))
- [Remote Store] Add support to reload repository metadata inplace ([#9569](https://github.com/opensearch-project/OpenSearch/pull/9569))
- [Metrics Framework] Add Metrics framework. ([#10241](https://github.com/opensearch-project/OpenSearch/pull/10241))
- Updating the separator for RemoteStoreLockManager since underscore is allowed in base64UUID url charset ([#10379](https://github.com/opensearch-project/OpenSearch/pull/10379))
- Add the means to extract the contextual properties from HttpChannel, TcpCChannel and TrasportChannel without excessive typecasting ([#10562](https://github.com/opensearch-project/OpenSearch/pull/10562))
- [Remote Store] Add Remote Store backpressure rejection stats to `_nodes/stats` ([#10524](https://github.com/opensearch-project/OpenSearch/pull/10524))
- [BUG] Fix java.lang.SecurityException in repository-gcs plugin ([#10642](https://github.com/opensearch-project/OpenSearch/pull/10642))
- Add telemetry tracer/metric enable flag and integ test. ([#10395](https://github.com/opensearch-project/OpenSearch/pull/10395))

### Deprecated

### Removed
- Remove spurious SGID bit on directories ([#9447](https://github.com/opensearch-project/OpenSearch/pull/9447))

### Fixed
- Fix ignore_missing parameter has no effect when using template snippet in rename ingest processor ([#9725](https://github.com/opensearch-project/OpenSearch/pull/9725))
- Fix broken backward compatibility from 2.7 for IndexSorted field indices ([#10045](https://github.com/opensearch-project/OpenSearch/pull/10045))
- Fix concurrent search NPE when track_total_hits, terminate_after and size=0 are used ([#10082](https://github.com/opensearch-project/OpenSearch/pull/10082))
- Fix remove ingest processor handing ignore_missing parameter not correctly ([10089](https://github.com/opensearch-project/OpenSearch/pull/10089))
- Fix failure in dissect ingest processor parsing empty brackets ([#9225](https://github.com/opensearch-project/OpenSearch/pull/9255))
- Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API ([#10101](https://github.com/opensearch-project/OpenSearch/pull/10101))
- Fix circular dependency in Settings initialization ([10194](https://github.com/opensearch-project/OpenSearch/pull/10194))
- Fix registration and initialization of multiple extensions ([10256](https://github.com/opensearch-project/OpenSearch/pull/10256))
- Fix Segment Replication ShardLockObtainFailedException bug during index corruption ([10370](https://github.com/opensearch-project/OpenSearch/pull/10370))
- Fix some test methods in SimulatePipelineRequestParsingTests never run and fix test failure ([#10496](https://github.com/opensearch-project/OpenSearch/pull/10496))

### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.12...2.x
3 changes: 2 additions & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,5 @@ jzlib = 1.1.3
resteasy = 6.2.4.Final

# opentelemetry dependencies
opentelemetry = 1.30.1
opentelemetry = 1.31.0
opentelemetrysemconv = 1.21.0-alpha
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,10 @@ public Map<String, String> parse(String inputString) {
int lookAheadMatches;
// start walking the input string byte by byte, look ahead for matches where needed
// if a match is found jump forward to the end of the match
for (; i < input.length; i++) {
while (i < input.length) {
// start is only used to record the value of i
int start = i;

lookAheadMatches = 0;
// potential match between delimiter and input string
if (delimiter.length > 0 && input[i] == delimiter[0]) {
Expand Down Expand Up @@ -283,8 +286,14 @@ public Map<String, String> parse(String inputString) {
delimiter = dissectPair.getDelimiter().getBytes(StandardCharsets.UTF_8);
// i is always one byte after the last found delimiter, aka the start of the next value
valueStart = i;
} else {
i++;
}
} else {
i++;
}
// i should change anyway
assert (i != start);
}
// the last key, grab the rest of the input (unless consecutive delimiters already grabbed the last key)
// and there is no trailing delimiter
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,12 @@

import org.opensearch.common.annotation.ExperimentalApi;

import java.io.Closeable;

/**
* Interface for metrics telemetry providers
*
* @opensearch.experimental
*/
@ExperimentalApi
public interface MetricsTelemetry extends MetricsRegistry, Closeable {
public interface MetricsTelemetry extends MetricsRegistry {

}
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@ public SpanScope withSpanInScope(Span span) {
return DefaultSpanScope.create(span, tracerContextStorage).attach();
}

@Override
public boolean isRecording() {
return true;
}

private Span createSpan(SpanCreationContext spanCreationContext, Span parentSpan) {
return tracingTelemetry.createSpan(spanCreationContext, parentSpan);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,10 @@ public interface Tracer extends HttpTracer, Closeable {
*/
SpanScope withSpanInScope(Span span);

/**
* Tells if the traces are being recorded or not
* @return boolean
*/
boolean isRecording();

}
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ public SpanScope withSpanInScope(Span span) {
return SpanScope.NO_OP;
}

@Override
public boolean isRecording() {
return false;
}

@Override
public void close() {

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ public void testCreateSpan() {

String spanName = defaultTracer.getCurrentSpan().getSpan().getSpanName();
assertEquals("span_name", spanName);
assertTrue(defaultTracer.isRecording());
}

@SuppressWarnings("unchecked")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,4 +155,28 @@ public void testNullValueWithOutIgnoreMissing() {
IngestDocument ingestDocument = new IngestDocument(originalIngestDocument);
expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
}

public void testMatchEmptyBrackets() {
IngestDocument ingestDocument = new IngestDocument(
"_index",
"_id",
null,
null,
null,
Collections.singletonMap("message", "[foo],[bar],[]")
);
DissectProcessor dissectProcessor = new DissectProcessor("", null, "message", "[%{a}],[%{b}],[%{c}]", "", true);
dissectProcessor.execute(ingestDocument);
assertEquals("foo", ingestDocument.getFieldValue("a", String.class));
assertEquals("bar", ingestDocument.getFieldValue("b", String.class));
assertEquals("", ingestDocument.getFieldValue("c", String.class));

ingestDocument = new IngestDocument("_index", "_id", null, null, null, Collections.singletonMap("message", "{}{}{}{baz}"));
dissectProcessor = new DissectProcessor("", null, "message", "{%{a}}{%{b}}{%{c}}{%{d}}", "", true);
dissectProcessor.execute(ingestDocument);
assertEquals("", ingestDocument.getFieldValue("a", String.class));
assertEquals("", ingestDocument.getFieldValue("b", String.class));
assertEquals("", ingestDocument.getFieldValue("c", String.class));
assertEquals("baz", ingestDocument.getFieldValue("d", String.class));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,38 @@ teardown:
}
]
}
---
"Test dissect processor can match empty brackets":
- do:
ingest.put_pipeline:
id: "my_pipeline"
body: >
{
"description": "_description",
"processors": [
{
"dissect" : {
"field" : "message",
"pattern" : "[%{a}][%{b}][%{c}]"
}
}
]
}
- match: { acknowledged: true }

- do:
index:
index: test
id: 1
pipeline: "my_pipeline"
body: {message: "[foo][bar][]"}

- do:
get:
index: test
id: 1
- match: { _source.message: "[foo][bar][]" }
- match: { _source.a: "foo" }
- match: { _source.b: "bar" }
- match: { _source.c: "" }
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,11 @@

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.shiro.SecurityUtils;
import org.apache.shiro.authc.AuthenticationToken;
import org.apache.shiro.authc.UsernamePasswordToken;
import org.opensearch.common.Randomness;
import org.opensearch.identity.IdentityService;
import org.opensearch.identity.Subject;
import org.opensearch.identity.noop.NoopSubject;
import org.opensearch.identity.tokens.AuthToken;
import org.opensearch.identity.tokens.BasicAuthToken;
import org.opensearch.identity.tokens.OnBehalfOfClaims;
Expand Down Expand Up @@ -88,20 +86,6 @@ public AuthToken issueServiceAccountToken(String audience) {
return token;
}

@Override
public Subject authenticateToken(AuthToken authToken) {
return new NoopSubject();
}

public boolean validateToken(AuthToken token) {
if (token instanceof BasicAuthToken) {
final BasicAuthToken basicAuthToken = (BasicAuthToken) token;
return basicAuthToken.getUser().equals(SecurityUtils.getSubject().toString())
&& basicAuthToken.getPassword().equals(shiroTokenPasswordMap.get(basicAuthToken));
}
return false;
}

public String getTokenInfo(AuthToken token) {
if (token instanceof BasicAuthToken) {
final BasicAuthToken basicAuthToken = (BasicAuthToken) token;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,6 @@ public void testShouldFailGetTokenInfo() {
assertThrows(UnsupportedAuthenticationToken.class, () -> shiroAuthTokenHandler.getTokenInfo(bearerAuthToken));
}

public void testShouldFailValidateToken() {
final BearerAuthToken bearerAuthToken = new BearerAuthToken("header.payload.signature");
assertFalse(shiroAuthTokenHandler.validateToken(bearerAuthToken));
}

public void testShoudPassMapLookupWithToken() {
final BasicAuthToken authToken = new BasicAuthToken("Basic dGVzdDp0ZTpzdA==");
shiroAuthTokenHandler.getShiroTokenPasswordMap().put(authToken, "te:st");
Expand Down
4 changes: 2 additions & 2 deletions plugins/repository-gcs/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ dependencies {
runtimeOnly "com.google.guava:guava:${versions.guava}"
api 'com.google.guava:failureaccess:1.0.1'

api 'com.google.http-client:google-http-client:1.43.2'
api 'com.google.http-client:google-http-client-appengine:1.43.2'
api 'com.google.http-client:google-http-client:1.43.3'
api 'com.google.http-client:google-http-client-appengine:1.43.3'
api 'com.google.http-client:google-http-client-gson:1.43.3'
api 'com.google.http-client:google-http-client-jackson2:1.43.3'

Expand Down

This file was deleted.

Loading

0 comments on commit 0953f54

Please sign in to comment.