Spike: Test OTel Manual instrumentation with NR OTel Span Prototype

[jasonjkeller]

Test whether the OTel Span Prototype captures spans from OTel manual instrumentation directly added via OTel APIs. What happens when instrumentation is generated by both OTel manual Instrumentation and a NR Java agent instrumentation (weave module/custom instrumentation)?

[workato-integration]

https://new-relic.atlassian.net/browse/NR-316385

[jasonjkeller]

Test app here: https://github.com/jasonjkeller/otel-api-test

Depending on the SpanKind the OTel Span Prototype will produce different results.

[jasonjkeller]

SpanKind.INTERNAL

Creating a span with no SpanKind, which defaults to SpanKind.INTERNAL, will not start a transaction:

    // create span with no span kind
    public static void noSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("noSpanKind").startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called noSpanKind");
            Thread.sleep(1000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

If SpanKind.INTERNAL spans occur within an already existing New Relic transaction they will be included in the trace:

    @Trace(dispatcher = true)
    public static void nrTraceNoSpanKind() {
        System.out.println("called nrTraceNoSpanKind");
        try {
            noSpanKind();
            Thread.sleep(500);
        } catch (InterruptedException ignored) {
        }
    }

---

[jasonjkeller]

SpanKind.CLIENT

Creating a span with SpanKind.CLIENT will not start a transaction. If a CLIENT span has certain db attributes it will be treated as a DB span, and other specific attributes will cause it to be treated as an external span:

    // create generic client span
    public static void clientSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("clientSpanKind").setSpanKind(SpanKind.CLIENT).startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called clientSpanKind");
            Thread.sleep(2000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

    // create DB client span
    public static void dbClientSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("owners select").setSpanKind(SpanKind.CLIENT)
                .setAttribute("db.system", "mysql")
                .setAttribute("db.operation", "select")
                .setAttribute("db.sql.table", "owners")
                .setAttribute("db.statement", "SELECT * FROM owners WHERE ssn = 4566661792").startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called dbClientSpanKind");
            Thread.sleep(2000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

    // create external client span
    public static void externalClientSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("example.com").setSpanKind(SpanKind.CLIENT)
                .setAttribute("server.address", "www.foo.bar")
                .setAttribute("url.full", "https://www.foo.bar:8080/search?q=OpenTelemetry#SemConv")
                .setAttribute("server.port", 8080)
                .setAttribute("http.request.method", "GET")
                .startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called externalClientSpanKind");
            Thread.sleep(2000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

If SpanKind.CLIENT spans occur within an already existing New Relic transaction they will be included in the trace:

    @Trace(dispatcher = true)
    public static void nrTraceClientSpanKind() {
        System.out.println("called nrTraceClientSpanKind");
        try {
            clientSpanKind();
            dbClientSpanKind();
            externalClientSpanKind();
            Thread.sleep(500);
        } catch (InterruptedException ignored) {
        }
    }

---

[jasonjkeller]

SpanKind.SERVER

Creating a span with SpanKind.SERVER will start a WebTransaction/Uri/* transaction. If SpanKind.SERVER spans occur within an already existing New Relic transaction they will be included in the trace:

    // create generic server span
    public static void serverSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("serverSpanKind").setSpanKind(SpanKind.SERVER)
                .setAttribute("url.path", "/whatever")
                .startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called serverSpanKind");
            Thread.sleep(1500);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

---

[jasonjkeller]

SpanKind.CONSUMER

Creating a span with SpanKind.CONSUMER will start a OtherTransaction/* transaction. If SpanKind.CONSUMER spans occur within an already existing New Relic transaction they will be included in the trace:

    // create generic consumer span
    public static void consumerSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("consumerSpanKind").setSpanKind(SpanKind.CONSUMER).startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called consumerSpanKind");
            Thread.sleep(1000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

---

---

There does seem to be some weirdness in the UI that will need to be investigated when transactions are started by SpanKind.CONSUMER span.

---

[jasonjkeller]

SpanKind.PRODUCER

Creating a span with SpanKind.PRODUCER will not start a transaction (Why not, given that a SpanKind.CONSUMER span does?).

    // create generic producer span
    public static void producerSpanKind() throws InterruptedException {
        Span span = tracer.spanBuilder("producerSpanKind").setSpanKind(SpanKind.PRODUCER).startSpan();
        try (Scope scope = span.makeCurrent()) {
            System.out.println("called producerSpanKind");
            Thread.sleep(1000);
        } catch (Throwable t) {
            span.recordException(t);
            throw t;
        } finally {
            span.end();
        }
    }

If SpanKind.PRODUCER spans occur within an already existing New Relic transaction they will be included in the trace (though it's effectively no different than a SpanKind.INTERNAL span):

---

[jasonjkeller]

Summary

Most things work as expected but there are two oddities:

1. Why is there no explicit support for SpanKind.PRODUCER spans?
2. Why do transactions started by SpanKind.CONSUMER spans seem to break the Transactions UI (as detailed in the above screenshot)?