Fastify auto instrumentation metrics missing http url/route

[nathanscully]

What version of OpenTelemetry are you using?

@opentelemetry/auto-instrumentations-node 0.43.0                                                    
@opentelemetry/exporter-metrics-otlp-proto 0.49.1                                                    
@opentelemetry/exporter-trace-otlp-proto 0.49.1                                               
@opentelemetry/sdk-metrics 1.22.0                                           
@opentelemetry/sdk-node 0.49.1                                           
@opentelemetry/sdk-trace-node 1.22.0
@opentelemetry/api 1.8.0 

What version of Node are you using?

20.11.1

What did you do?

I've attempted to instrument otel using the autoinstrumentation in a Fastify server:

import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { NodeSDK } from "@opentelemetry/sdk-node";

const metricReader = new PeriodicExportingMetricReader({
  exportIntervalMillis: 10000,
  exporter: new OTLPMetricExporter(),
});

const traceExporter = new OTLPTraceExporter();

const sdk = new NodeSDK({
  traceExporter: traceExporter,
  instrumentations: [getNodeAutoInstrumentations()],
  metricReader: metricReader,
  serviceName: "server",
});
export default sdk;

What did you expect to see?

I am using the grafana-agent to collect the otel data and emit it to Grafana Cloud. This integration seems to be mostly working but I expected more metrics. I can see traces + logs, but the metrics are all grouped together by status code and method, i.e.GET 200. These are the only metrics:

http_client_duration_milliseconds_bucket
http_client_duration_milliseconds_count
http_client_duration_milliseconds_sum
http_server_duration_milliseconds_bucket
http_server_duration_milliseconds_count
http_server_duration_milliseconds_sum

Within those metrics the only labels are:

http_flavour
http_method
http_scheme
http_status_code

Which results in:

{__name__="http_server_duration_milliseconds_sum", http_flavor="1.1", http_method="GET", http_scheme="http", http_status_code="200", job="server", net_host_name="localhost", net_host_port="8080"}
There is no field to pivot by url or route.

What did you see instead?

I would have expected to see metrics by http_route so I can see how many requests per route, per status. I thought it might be related to #1569 but maybe I've misunderstood what comes out of the box with auto-instrumentation.

If I need to do some extra for my configuration to see typical RED metrics (Rate, Errors, Duration) per URL endpoint can anyone point me in the right direction. Preferably using a middleware or similar so I don't need to manually wire up each endpoint.

[pichlermarc]

High-cardinality data is a large concern for metrics, this is why we try to avoid url. For a server, url may include IDs (route /foo/{id} can be /foo/123, foo/234, foo/345, ...). For cumulative metrics (e.g. Prometheus), we also have the keep each metrics stream (unique combination of attributes) in memory, which can in the worst case cause the instrumented app to run out of memory.

It is not the case for spans and logs, as everything can be safely discarded once it's exported.

That being said, route should be populated for metrics as it's not a high-cardinality attribute. Can you verify that http.route is present on your spans? If it is present on your server spans, it should also be present on the server metrics.

[nathanscully]

Thanks for clarifying and writing up the response. I'm closing this as I'll put it under 'implementation error' after digging into it further.

Looks like my code was not loading the fastify instrumentation correctly as I'm using ESM. See open-telemetry/opentelemetry-js#4553 - I ended up using a work around someone else suggested to use the jiti loader as a temporary alternative and its loading properly now.