Add request tracing framework #7648

suranjay · 2023-05-22T02:20:32Z

Description

This change adds request tracing framework in OpenSearch. It also integrates Opentelemetry with OpenSearch as it uses the framework for implementing tracing capabilities in OpenSearch.

The tracing framework automatically handles context propagation between different threads, tasks and nodes.

Note: The integrations tests will be covered in the subsequent PR to limit the changes in this PR.

Related Issues

Resolves #7543, #7544

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-22T02:48:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15830/
CommitID: ec4802b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-24T08:09:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/16161/
CommitID: 5af55b1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja

Thanks @suranjay picking this up. What is the overhead of Tracer framework disabled by default? Do we have some nos.?

server/src/main/java/org/opensearch/tracing/DefaultTracer.java

server/src/main/java/org/opensearch/common/util/concurrent/ThreadContext.java

server/src/main/java/org/opensearch/tracing/OTelResourceProvider.java

server/src/main/java/org/opensearch/tracing/OSSpan.java

server/src/main/java/org/opensearch/tracing/DefaultTracer.java

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

server/src/main/java/org/opensearch/tracing/OSSpan.java

server/src/main/java/org/opensearch/tracing/OTelResourceProvider.java

Bukhtawar

Please consider re-wiring dependencies. It seems too tightly coupled with Otel. Please also consider moving this to a module

github-actions · 2023-05-28T15:46:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/16431/
CommitID: cb9a32a
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-05-28T15:46:57Z

Codecov Report

Merging #7648 (7591c1f) into main (4c02dd1) will increase coverage by 0.07%.
The diff coverage is 77.25%.

@@             Coverage Diff              @@
##               main    #7648      +/-   ##
============================================
+ Coverage     70.90%   70.97%   +0.07%     
- Complexity    56710    56866     +156     
============================================
  Files          4733     4752      +19     
  Lines        268333   268599     +266     
  Branches      39322    39336      +14     
============================================
+ Hits         190254   190645     +391     
+ Misses        61996    61888     -108     
+ Partials      16083    16066      -17

Impacted Files	Coverage Δ
...rg/opensearch/telemetry/tracing/SpanReference.java	`0.00% <0.00%> (ø)`
...pensearch/common/settings/FeatureFlagSettings.java	`50.00% <ø> (ø)`
...racing/ThreadContextBasedTracerContextStorage.java	`16.66% <16.66%> (ø)`
.../opensearch/telemetry/tracing/noop/NoopTracer.java	`25.00% <25.00%> (ø)`
server/src/main/java/org/opensearch/node/Node.java	`84.95% <27.27%> (-0.89%)`	⬇️
...pensearch/telemetry/tracing/NoopTracerFactory.java	`50.00% <50.00%> (ø)`
...elemetry/tracing/OTelTracingContextPropagator.java	`75.00% <75.00%> (ø)`
...va/org/opensearch/telemetry/TelemetrySettings.java	`75.00% <75.00%> (ø)`
...search/telemetry/tracing/OTelTracingTelemetry.java	`82.35% <82.35%> (ø)`
...rg/opensearch/telemetry/tracing/OTelTelemetry.java	`83.33% <83.33%> (ø)`
... and 14 more

... and 457 files with indirect coverage changes

github-actions · 2023-05-29T09:06:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: 764862b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-29T09:20:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: 60c2a47
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-29T09:29:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: 7f5a378
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: suranjay <[email protected]>

github-actions · 2023-05-29T09:35:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: f275db8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-29T09:35:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: f275db8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: suranjay <[email protected]>

github-actions · 2023-05-29T10:43:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: 0d67058
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

suranjay · 2023-05-29T11:57:32Z

Thanks @suranjay picking this up. What is the overhead of Tracer framework disabled by default? Do we have some nos.?

@shwetathareja There is no overhead with of Tracer framework in disabled state. We have verified this by doing performance benchmarking using OSB

Signed-off-by: suranjay <[email protected]>

github-actions · 2023-05-29T15:10:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL: null
CommitID: a2dd91d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja

Thanks @suranjay . I am still going over the changes, published some feedback after first pass. Btw, you can change the status to actual PR from DRAFT.

buildSrc/version.properties

modules/tracer-otel/src/main/java/org/opensearch/tracing/DefaultTracer.java

modules/tracer-otel/src/main/java/org/opensearch/tracing/OTelResourceProvider.java

modules/tracer-otel/src/main/java/org/opensearch/tracing/DefaultTracer.java

server/src/main/java/org/opensearch/tracing/TracerModule.java

server/src/main/java/org/opensearch/tracing/SpanHolder.java

server/build.gradle

modules/tracer-otel/src/main/java/org/opensearch/tracing/DefaultTracer.java

gashutos

Few files left from TracerManager.....

modules/tracer-otel/build.gradle

modules/tracer-otel/src/main/java/org/opensearch/tracing/DefaultSpan.java

modules/tracer-otel/src/main/java/org/opensearch/tracing/DefaultTracer.java

modules/tracer-otel/src/main/plugin-metadata/plugin-security.policy

modules/tracer-otel/src/test/java/org/opensearch/tracing/exporter/MockAppender.java

server/src/main/java/org/opensearch/common/util/FeatureFlags.java

server/src/main/java/org/opensearch/tracing/Level.java

server/src/main/java/org/opensearch/tracing/Span.java

github-actions · 2023-06-27T16:21:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18473/
CommitID: c77b09a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-27T16:21:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18472/
CommitID: bae04ac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-27T16:26:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18474/
CommitID: c77b09a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/common/util/concurrent/ThreadContextStatePropagator.java

rishabhmaurya · 2023-06-27T16:34:36Z

Thanks @suranjay, LGTM!

reta · 2023-06-27T16:41:07Z

Thanks @suranjay !

reta · 2023-06-27T16:41:49Z

(PS: the build should be fixed by #8290)

github-actions · 2023-06-28T03:56:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18533/
CommitID: 3e1e031
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-28T04:06:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreRemoteStoreIndicesWithoutRemoteTranslog
      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreRemoteStoreIndicesWithRemoteTranslog
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18534/
CommitID: 3e1e031
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: suranjay <[email protected]>

github-actions · 2023-06-28T05:52:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18540/
CommitID: 7637b37
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-28T06:11:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreRemoteStoreIndicesWithoutRemoteTranslog
      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreRemoteStoreIndicesWithRemoteTranslog
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18542/
CommitID: 7591c1f
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-06-28T06:45:39Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7648-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 03bc19220a711853660652a4761c05285f597884
# Push it to GitHub
git push --set-upstream origin backport/backport-7648-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7648-to-2.x.

* Add request tracing framework Signed-off-by: suranjay <[email protected]> * Introduce ThreadContextStatePropagator to propagate request and transient headers within ThreadContext Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: suranjay <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Andriy Redko <[email protected]>

* Add request tracing framework Signed-off-by: suranjay <[email protected]> * Introduce ThreadContextStatePropagator to propagate request and transient headers within ThreadContext Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: suranjay <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

suranjay changed the title ~~Add request tracing framework~~ [WIP] Add request tracing framework May 22, 2023

suranjay mentioned this pull request May 22, 2023

[Prototype] Distributed Tracing #7026

Open

shwetathareja reviewed May 24, 2023

View reviewed changes

server/src/main/java/org/opensearch/tracing/DefaultTracer.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/tracing/DefaultTracer.java Outdated Show resolved Hide resolved

monusingh-1 reviewed May 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java Outdated Show resolved Hide resolved

Bukhtawar reviewed May 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/tracing/OSSpan.java Outdated Show resolved Hide resolved

Bukhtawar reviewed May 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/tracing/OTelResourceProvider.java Outdated Show resolved Hide resolved

Bukhtawar reviewed May 25, 2023

View reviewed changes

suranjay force-pushed the main-opentelemetry branch from 764862b to 60c2a47 Compare May 29, 2023 09:19

suranjay closed this May 29, 2023

suranjay force-pushed the main-opentelemetry branch from 60c2a47 to 7f5a378 Compare May 29, 2023 09:29

suranjay added 2 commits May 29, 2023 15:02

Add request tracing framework

53ca1dc

Signed-off-by: suranjay <[email protected]>

Extract Otel dependencies to a separate module

f275db8

Signed-off-by: suranjay <[email protected]>

suranjay reopened this May 29, 2023

Fix javadocs error

0d67058

Signed-off-by: suranjay <[email protected]>

Add request tracing framework

a2dd91d

Signed-off-by: suranjay <[email protected]>

shwetathareja reviewed May 29, 2023

View reviewed changes

gashutos reviewed May 29, 2023

View reviewed changes

rishabhmaurya reviewed Jun 27, 2023

View reviewed changes

server/src/main/java/org/opensearch/common/util/concurrent/ThreadContextStatePropagator.java Show resolved Hide resolved

rishabhmaurya approved these changes Jun 27, 2023

View reviewed changes

reta approved these changes Jun 27, 2023

View reviewed changes

Remove endSpan method

7637b37

Signed-off-by: suranjay <[email protected]>

suranjay force-pushed the main-opentelemetry branch from 3e1e031 to 7637b37 Compare June 28, 2023 05:26

merge changes from main

7591c1f

Signed-off-by: suranjay <[email protected]>

shwetathareja approved these changes Jun 28, 2023

View reviewed changes

shwetathareja merged commit 03bc192 into opensearch-project:main Jun 28, 2023

shwetathareja added the backport 2.x Backport to 2.x branch label Jun 28, 2023

Gaganjuneja mentioned this pull request Jun 29, 2023

Adds mock implementation for TelemetryPlugin #8357

Merged

6 tasks

rishabhmaurya mentioned this pull request Aug 1, 2023

Trace events lib changes #8816

Closed

9 tasks

sohami mentioned this pull request Aug 2, 2023

[RFC] Admission Controller framework for OpenSearch #8910

Open

Gaganjuneja mentioned this pull request Aug 2, 2023

Adds support for tracing runnable scenarios #8831

Merged

6 tasks

This was referenced Aug 17, 2023

Add instrumentation so that SQL plugin sends telemetry traces? Bit-Quill/opentelemetry-collector-contrib#24

Closed

[FEATURE] Add instrumentation to SQL plugin for telemetry traces opensearch-project/sql#1983

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add request tracing framework #7648

Add request tracing framework #7648

suranjay commented May 22, 2023 •

edited

Loading

github-actions bot commented May 22, 2023

github-actions bot commented May 24, 2023

shwetathareja left a comment

Bukhtawar left a comment

github-actions bot commented May 28, 2023

codecov bot commented May 28, 2023 •

edited

Loading

github-actions bot commented May 29, 2023

github-actions bot commented May 29, 2023

github-actions bot commented May 29, 2023

github-actions bot commented May 29, 2023

github-actions bot commented May 29, 2023

github-actions bot commented May 29, 2023

suranjay commented May 29, 2023

github-actions bot commented May 29, 2023

shwetathareja left a comment

gashutos left a comment

github-actions bot commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

rishabhmaurya commented Jun 27, 2023

reta commented Jun 27, 2023

reta commented Jun 27, 2023

github-actions bot commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

opensearch-trigger-bot bot commented Jun 28, 2023

Add request tracing framework #7648

Add request tracing framework #7648

Conversation

suranjay commented May 22, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 24, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja left a comment

Choose a reason for hiding this comment

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented May 28, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented May 28, 2023 • edited Loading

Codecov Report

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

suranjay commented May 29, 2023

github-actions bot commented May 29, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja left a comment

Choose a reason for hiding this comment

gashutos left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 27, 2023

Gradle Check (Jenkins) Run Completed with:

rishabhmaurya commented Jun 27, 2023

reta commented Jun 27, 2023

reta commented Jun 27, 2023

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jun 28, 2023

suranjay commented May 22, 2023 •

edited

Loading

codecov bot commented May 28, 2023 •

edited

Loading