Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTel span support #1886

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b108750
Add OTel span support
sdaubin May 3, 2024
53a0a96
Documentation
sdaubin May 3, 2024
50ada7a
Add static constructor
sdaubin May 3, 2024
fe873d7
Add test showing the SpanBuilder.setParent doesn't link async work
sdaubin May 3, 2024
8faee60
Implement more of the OTel API for spans
sdaubin May 6, 2024
583a36a
Add a switch to disable custom otel span builder
sdaubin May 7, 2024
bcb9146
Merge remote-tracking branch 'origin' into saxon/replace-otel-spans2
sdaubin May 7, 2024
c7bdc0a
Fix test
sdaubin May 8, 2024
4563453
Log if `otel.exporter.otlp.endpoint` is set
sdaubin May 8, 2024
224947c
Update DefaultTracerTest.java
sdaubin May 8, 2024
6041d13
Allow individual opentelemetry instrumentation scopes to be disabled
sdaubin May 10, 2024
a9029cc
Try to fix test
sdaubin May 10, 2024
67efb93
Use OpenTelemetry no op Span instance
sdaubin May 16, 2024
509ce8a
Fix sql obfuscation of QueryConverter used by otel spans
sdaubin May 21, 2024
4911fc1
Add another external span test
sdaubin May 21, 2024
ba4835a
Merge branch 'main' into saxon/replace-otel-spans2
jasonjkeller Jun 28, 2024
972a16c
Add copyright header and formatting
jasonjkeller Jul 1, 2024
12e2ef4
Merge branch 'main' of github.com:newrelic/newrelic-java-agent into s…
jasonjkeller Sep 23, 2024
f99b9ed
Merge branch 'main' of github.com:newrelic/newrelic-java-agent into s…
jasonjkeller Oct 7, 2024
35c91f8
Add some comments
jasonjkeller Oct 29, 2024
0750c0b
Merge main
jasonjkeller Nov 14, 2024
c308a60
Add readme for OTel functionality
jasonjkeller Dec 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

package com.newrelic.agent.bridge;

import com.newrelic.api.agent.Token;

import java.lang.reflect.InvocationHandler;


Expand All @@ -19,10 +21,19 @@ public interface ExitTracer extends InvocationHandler, TracedMethod {
*/
void finish(int opcode, Object returnValue);

default void finish() {
// 177 is Opcodes.RETURN
finish(177, null);
}

/**
* Called when a method invocation throws an exception.
*
* @param throwable
*/
void finish(Throwable throwable);

default Token getToken() {
return NoOpToken.INSTANCE;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@

import com.newrelic.api.agent.NewRelic;
import com.newrelic.api.agent.Token;
import com.newrelic.api.agent.Trace;

public interface Instrumentation {

Expand Down Expand Up @@ -59,6 +58,10 @@ ExitTracer createTracer(Object invocationTarget, int signatureId, boolean dispat

ExitTracer createScalaTxnTracer();

default ExitTracer createTracer(String metricName, int flags) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The createSegment API still generates an asynchronous tracer, which is not what we want. And from the bridge there's no easy way to register a signature to get a signatureId. This was the easiest way to get a normal, synchronous tracer.

return null;
}

/**
* Returns the current transaction. This should not be called directly - instead use {@link Agent#getTransaction()}.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
*/
public interface TracedMethod extends com.newrelic.api.agent.TracedMethod {

default String getTraceId() {
return "0000000000000000";
}
default String getSpanId() {
return "0000000000000000";
}
/**
* Returns the parent of this traced method, or null if this is the root tracer.
*
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
*
* * Copyright 2024 New Relic Corporation. All rights reserved.
* * SPDX-License-Identifier: Apache-2.0
*
*/

package com.newrelic.agent.bridge.datastore;

import com.newrelic.api.agent.QueryConverter;

public final class SqlQueryConverter implements QueryConverter<String> {
public static final QueryConverter<String> INSTANCE = new SqlQueryConverter();

private SqlQueryConverter() {
}

@Override
public String toRawQueryString(String rawQuery) {
return rawQuery;
}

@Override
public String toObfuscatedQueryString(String rawQuery) {
return null;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,6 @@ public interface SpanEvent {
String getStatusText();

Map<String, Object> getAgentAttributes();

Map<String, Object> getUserAttributes();
}
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,9 @@ public String getStatusText() {
public Map<String, Object> getAgentAttributes() {
return spanEvent.getAgentAttributes();
}

@Override
public Map<String, Object> getUserAttributes() {
return spanEvent.getUserAttributesCopy();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# OpenTelemetry Instrumentation

This instrumentation module weaves parts of the OpenTelemetry SDK to incorporate bits of OpenTelemetry functionality into the New Relic Java agent.

Specifically it can:
* Detect OpenTelemetry Spans and include them in New Relic Java agent traces.
* Detect OpenTelemetry dimensional metrics and report them to the APM entity being monitored by the Java agent.
* Autoconfigure the OpenTelemetry SDK so that OpenTelemetry data is sent to New Relic and properly associated with an APM entity guid.

## New Relic Java Agent Configuration

To use the OpenTelemetry Span and dimensional metric functionality incorporated into the New Relic Java agent you must enable the following config options:

Configuration via yaml:
```
opentelemetry:
sdk:
autoconfigure:
enabled: true
spans:
enabled: true
```

Configuration via system property:
```
-Dopentelemetry.sdk.autoconfigure.enabled=true
-Dopentelemetry.sdk.spans.enabled=true
```

Configuration via environment variable:
```
NEW_RELIC_OPENTELEMETRY_SDK_AUTOCONFIGURE_ENABLED=true
NEW_RELIC_OPENTELEMETRY_SDK_SPANS_ENABLED=true
```

## OpenTelemetry Dimensional Metrics

OpenTelemetry APIs can be used to create dimensional metrics which will be detected by the New Relic Java agent and reported to the APM entity being monitored by the New Relic Java agent.

To use this functionality, enable the feature as documented above, add the required `opentelemetry` dependencies to your application:
```groovy
implementation(platform("io.opentelemetry:opentelemetry-bom:1.44.1"))
implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure")
implementation("io.opentelemetry:opentelemetry-exporter-otlp")
```

Then utilize the OpenTelemetry APIs to record dimensional metrics:
```java
LongCounter longCounter = GlobalOpenTelemetry.get().getMeterProvider().get("my-application").counterBuilder("my.application.counter").build();
longCounter.add(1, Attributes.of(AttributeKey.stringKey("foo"), "bar"));
```

Any recorded dimensional metrics can be found in the Metrics Explorer for the associated APM entity and can be used to build custom dashboards.

## OpenTelemetry Spans

Documented below are several approaches for incorporating OpenTelemetry Spans into New Relic Java agent traces.

### `@WithSpan` Annotation

The New Relic Java agent will detect usage of the OpenTelemetry [@WithSpan](https://opentelemetry.io/docs/zero-code/java/agent/annotations/) annotation. The `@WithSpan` annotation can be used as an alternative to the `@Trace` annotation.

This does not currently support the following config options:
* [Suppressing @WithSpan instrumentation](https://opentelemetry.io/docs/zero-code/java/agent/annotations/#suppressing-withspan-instrumentation)
* [Creating spans around methods with otel.instrumentation.methods.include](https://opentelemetry.io/docs/zero-code/java/agent/annotations/#creating-spans-around-methods-with-otelinstrumentationmethodsinclude)

Note that OpenTelemetry config properties can be set through environment or system properties, like our agent, and eventually through a config file. We can use our existing OpenTelemetry instrumentation model to get access to the normalized version of the instrumentation settings to include and exclude methods and pass those to the core agent through the bridge.

See `ClassTransformerConfigImpl.java` for implementation details of the `@WithSpan` annotation.

### Spans Emitted From OpenTelemetry Instrumentation

The New Relic Java agent will detect Spans emitted by [OpenTelemetry instrumentation](https://opentelemetry.io/docs/languages/java/instrumentation/). It does this by weaving the `io.opentelemetry.sdk.trace.SdkTracerProvider` so that it will create a New Relic Tracer each time an OpenTelemetry Span is started and weaving the `io.opentelemetry.context.Context` to propagate context between New Relic and OpenTelemetry Spans.

Currently, the New Relic Java agent does not load any OpenTelemetry instrumentation it simply detects Spans emitted by OpenTelemetry manual instrumentation, native instrumentation, library instrumentation, or zero code instrumentation (i.e. bytecode instrumentation that would also require running the OpenTelemetry Java agent).

Depending on the OpenTelemetry Span `SpanKind`, it may result in the New Relic Java agent starting a transaction (when one doesn't already exist).

* `SpanKind.INTERNAL`
* Creating a span with no `SpanKind`, which defaults to `SpanKind.INTERNAL`, will not start a transaction
* If `SpanKind.INTERNAL` spans occur within an already existing New Relic transaction they will be included in the trace
* `SpanKind.CLIENT`
* Creating a span with `SpanKind.CLIENT` will not start a transaction. If a `CLIENT` span has certain db attributes it will be treated as a DB span, and other specific attributes will cause it to be treated as an external span
* If `SpanKind.CLIENT` spans occur within an already existing New Relic transaction they will be included in the trace
* `SpanKind.SERVER`
* Creating a span with `SpanKind.SERVER` will start a `WebTransaction/Uri/*` transaction.
* If `SpanKind.SERVER` spans occur within an already existing New Relic transaction they will be included in the trace
* `SpanKind.CONSUMER`
* Creating a span with `SpanKind.CONSUMER` will start a `OtherTransaction/*` transaction.
* If `SpanKind.CONSUMER` spans occur within an already existing New Relic transaction they will be included in the trace
* `SpanKind.PRODUCER`
* Creating a span with `SpanKind.PRODUCER` will not start a transaction. There is no explicit processing for `PRODUCER` spans currently.
* If `SpanKind.PRODUCER` spans occur within an already existing New Relic transaction they will be included in the trace (though it's effectively no different from a `SpanKind.INTERNAL` span)
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ dependencies {
implementation(project(":newrelic-weaver-api"))
implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.28.0")
testImplementation("junit:junit:4.12")
testImplementation("io.opentelemetry:opentelemetry-exporter-otlp:1.28.0")
testImplementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:1.28.0")
testImplementation("com.google.guava:guava:30.1.1-jre")
}

jar {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
*
* * Copyright 2024 New Relic Corporation. All rights reserved.
* * SPDX-License-Identifier: Apache-2.0
*
*/

package io.opentelemetry.context;

import com.newrelic.agent.bridge.AgentBridge;
import com.newrelic.agent.bridge.ExitTracer;
import com.newrelic.agent.bridge.Transaction;
import com.newrelic.api.agent.TracedMethod;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.sdk.trace.ExitTracerSpan;

/**
* Helper class for managing the OpenTelemetry Context
*/
class ContextHelper {
private ContextHelper() {
}

/**
* If there's no span on the context, but there is a NR tracer on the stack, return a context with our span.
*/
public static Context current(Context context) {
Span currentSpan = Span.fromContext(context);
if (currentSpan == Span.getInvalid()) {
Transaction transaction = AgentBridge.getAgent().getTransaction(false);
if (transaction != null) {
TracedMethod tracedMethod = transaction.getTracedMethod();
if (tracedMethod instanceof ExitTracer) {
return context.with(ExitTracerSpan.wrap((ExitTracer) tracedMethod));
}
}
}
return context;
}

/**
* If there's currently no NR transaction but the current contains a NR span, create a
* {@link com.newrelic.api.agent.Token} related to that span's transaction and hook it into
* the returned {@link Scope}.
*/
public static Scope makeCurrent(Context context, Scope scope) {
final Transaction currentTransaction = AgentBridge.getAgent().getTransaction(false);
if (currentTransaction == null) {
Span currentSpan = Span.fromContext(context);

if (currentSpan instanceof ExitTracerSpan) {
return ((ExitTracerSpan) currentSpan).createScope(scope);
}
}
return scope;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
*
* * Copyright 2024 New Relic Corporation. All rights reserved.
* * SPDX-License-Identifier: Apache-2.0
*
*/

package io.opentelemetry.context;

import com.newrelic.api.agent.weaver.MatchType;
import com.newrelic.api.agent.weaver.Weave;
import com.newrelic.api.agent.weaver.Weaver;

/**
* Weaved to manage the OpenTelemetry Context
*/
@Weave(type = MatchType.Interface, originalName = "io.opentelemetry.context.Context")
public abstract class Context_Instrumentation {
public static Context current() {
return ContextHelper.current(Weaver.callOriginal());
}

public Scope makeCurrent() {
return ContextHelper.makeCurrent((Context) this, Weaver.callOriginal());
}
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
/*
*
* * Copyright 2024 New Relic Corporation. All rights reserved.
* * SPDX-License-Identifier: Apache-2.0
*
*/

package io.opentelemetry.sdk.autoconfigure;

import com.newrelic.api.agent.NewRelic;
Expand All @@ -7,16 +14,20 @@

import java.util.logging.Level;

/**
* Weaved to autoconfigure the OpenTelemetrySDK properties
* and resources for compatability with New Relic.
*/
@Weave(type = MatchType.ExactClass)
public class AutoConfiguredOpenTelemetrySdk {

public static AutoConfiguredOpenTelemetrySdkBuilder builder() {
final AutoConfiguredOpenTelemetrySdkBuilder builder = Weaver.callOriginal();
Boolean autoConfigure = NewRelic.getAgent().getConfig().getValue("opentelemetry.sdk.autoconfigure.enabled");
final Boolean autoConfigure = NewRelic.getAgent().getConfig().getValue("opentelemetry.sdk.autoconfigure.enabled");
if (autoConfigure == null || autoConfigure) {
NewRelic.getAgent().getLogger().log(Level.INFO, "Appending OpenTelemetry SDK customizers");
builder.addPropertiesCustomizer(new PropertiesCustomizer());
builder.addResourceCustomizer(new ResourceCustomer());
builder.addPropertiesCustomizer(OpenTelemetrySDKCustomizer::applyProperties);
builder.addResourceCustomizer(OpenTelemetrySDKCustomizer::applyResources);
}
return builder;
}
Expand Down
Loading
Loading