JVM metrics not visible in New Relic APM due to missing service.instance.id

[ffsetit]

I have the OLTP plugin installed and configured to send data to New Relic which is working correctly.
However when looking at the APM data the JVM metrics are not displayed correctly with the following message displayed

The JVMs view lets you identify instances of your service that are behaving unusually and diagnose problems related to memory, garbage collection, or other internal runtime behavior. To see runtime/VM metrics broken down by instance, add the service.instance.id attribute to your OpenTelemetry metrics. To see all runtime/VM metrics, go to metrics explorer.

The raw data is present, however the service.instance.id attribute is not. I attempted to add this via the plugins advanced config as
service.instance.id=jenkins-test however upon restarting Jenkins the attribute is still not present and the startup logs contain
OpenTelemetry SDK initialized: SDK [config: otel.traces.exporter=otlp, otel.metrics.exporter=otlp, otel.exporter.otlp.endpoint=https://otlp.nr-data.net, resource: service.name=jenkins, service.namespace=jenkins, service.version=2.391]
indicating that the additional SDK parameter was not read.

Jenkins: 2.391
OLTP Plugin: 2.11.0

[pnvnd]

@ffsetit I tested with the Jenkins OpenTelemetry Plugin 2.11.0 and Jenkins 2.375.3 and was able to see the JVM metrics in New Relic after setting the environment variable OTEL_RESOURCE_ATTRIBUTES="service.instance.id=localhost"

For reference, I did this with Docker using the following:

docker run --name jenkinsci -d -e OTEL_LOGS_EXPORTER="otlp" -e OTEL_RESOURCE_ATTRIBUTES="service.instance.id=localhost" -p 8080:8080 jenkins/jenkins:lts

In New Relic, it looks like this:

Clicking on JVM and on localhost I see these JVM metrics:

Let me know if this works for you.

[ffsetit]

@pnvnd i can confirm that adding Environment=OTEL_RESOURCE_ATTRIBUTES=service.instance.id=..... to the systemd override.conf file does add the attribute and jvm metrics become visible in new relic.

Is this a bug that adding the config to manage jenkins > configure system > OpenTelemetry (advanced) > configuration properties did not work?

[cyrille-leclerc]

Thanks for the great feedback. We will fix this soon.
Note that the service.instance.id attribute is being clarified at the moment, this will make our work easier. See:

• Mark service and telemetry.sdk resource attributes as stable. open-telemetry/opentelemetry-specification#3202

Is this a bug that adding the config to manage jenkins > configure system > OpenTelemetry (advanced) > configuration properties did not work?

Good catch. It's not supported by the code and it would be good to support it.

See:

• https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-2.11.0/src/main/java/io/jenkins/plugins/opentelemetry/OpenTelemetrySdkProvider.java#L136
• https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-2.11.0/src/main/java/io/jenkins/plugins/opentelemetry/OpenTelemetryConfiguration.java#L139

[cyrille-leclerc]

See

• Add service.instance.id resource attribute #603

[cyrille-leclerc]

Note that we are re-aligning with the OTel Java Instrumentation official JVM runtime metrics adopting the instrumentation:opentelemetry-runtime-metrics lib.

• Use official instrumentation:opentelemetry-runtime-metrics #611

[cyrille-leclerc]

Can you please test https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.12.0-rc1

It uses the instrumentation of the Otel Java Auto Instrumentation library to collect JVM / runtime metrics and it produces service.instance.id.