How do i view the traces exported to OTLP exporter

[kushal-ti]

I have a simple express app running locally, configured to export traces to the OTLP eXPORTED

const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const exporter = new OTLPTraceExporter({
  // optional - url default value is http://localhost:55681/v1/traces
  url: 'http://localhost:55681/v1/traces',
  // optional - collection of custom headers to be sent with each request, empty by default
  headers: {},
});
// For troubleshooting, set the log level to DiagLogLevel.DEBUG
// diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);

const sdk = new opentelemetry.NodeSDK({
  traceExporter: exporter,
  spanProcessor: new SimpleSpanProcessor(exporter),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

I've previously tried printing the traces to stdout using the console exported and have confirmed it works, however, now that i'm exporting to the OTLPExporter, I can't figure out how to actually view the traces

[fatsheep9146]

Do you deploy otel collector to receiver those traces? If so, what is the configuration file of the otel collector?

[kushal-ti]

I have an otel collector running locally on my machine, but i'm not sure how to configure it to receive the logs, like i said, haven't been able to find any examples on this

this is my current configuration file

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

  opencensus:

  # Collect own metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 10s
          static_configs:
            - targets: ["0.0.0.0:8888"]

  jaeger:
    protocols:
      grpc:
      thrift_binary:
      thrift_compact:
      thrift_http:

  zipkin:

processors:
  batch:

exporters:
  logging:
    logLevel: info

service:
  pipelines:
    traces:
      receivers: [otlp, opencensus, jaeger, zipkin]
      processors: [batch]
      exporters: [logging]

    metrics:
      receivers: [otlp, opencensus, prometheus]
      processors: [batch]
      exporters: [logging]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

  extensions: [health_check, pprof, zpages]

It's pretty much just the default config file you get after installing the deb package. Any help on this would be greatly appreciated

[kushal-ti]

also, there are all the port the otel collector is listening to, it includes the grpc and http endpoints of otel and the default port for exporting traces listed in the collector docs

tcp6       0      0 :::14268                :::*                    LISTEN      7824/otelcol        
tcp6       0      0 :::4317                 :::*                    LISTEN      7824/otelcol        
tcp6       0      0 :::4318                 :::*                    LISTEN      7824/otelcol        
tcp6       0      0 :::55678                :::*                    LISTEN      7824/otelcol        
tcp6       0      0 :::55679                :::*                    LISTEN      7824/otelcol        
tcp6       0      0 :::9411                 :::*                    LISTEN      7824/otelcol           
tcp6       0      0 :::14250                :::*                    LISTEN      7824/otelcol

[fatsheep9146]

exporters:
  logging:
    logLevel: info

According to your config file, the exporter you used is logging exporter. And you can see the doc about logging exporter in here.

As the documents said

Exports data to the console via zap.Logger.

So in my opinion, the traces data that collector received should output to collector's stdout. You can check the stdout of your collector.

By the way, I read about the code details about how logging exporter consume the traces data.

opentelemetry-collector/exporter/loggingexporter/logging_exporter.go

Lines 42 to 54 in 277f98e

	func (s *loggingExporter) pushTraces(_ context.Context, td ptrace.Traces) error {
	s.logger.Info("TracesExporter", zap.Int("#spans", td.SpanCount()))
	if !s.logger.Core().Enabled(zapcore.DebugLevel) {
	return nil
	}
	buf, err := s.tracesMarshaler.MarshalTraces(td)
	if err != nil {
	return err
	}
	s.logger.Debug(string(buf))
	return nil
	}

I find that if loglevel is set to INFO, you can only see the content output to stdout just like this

TracesExporter #spans 10

So if you want to see the origin content of your trace data, you should change the configuration to

exporters:
  logging:
    loglevel: debug

I have not tested the above thought by myself, you can test as I suggest to clarify it :)

[kushal-ti]

Hi, i'm trying to confirm what you recommended, but facing some issues. Sorry if this is a really basic question, but how do i view the stdout of the otel collector process. The generic method of tail -f /proc/<pid>/fd/1 doesn't work, throwing this error

     tail: cannot open '/proc/19019/fd/1' for reading: No such device or address

further more, looking at the actual files inside of /proc/<pid>/fd i see that 1 is a socket

lrwx------ 1 otel otel 64 Jun 14 11:39 0 -> /dev/null
lrwx------ 1 otel otel 64 Jun 14 11:39 1 -> 'socket:[166089]'

I'm not sure what to do here.
Additionally, after i tried to update the config file, to have jaeger as the exporter instead of logging and tried to do

sudo systemctl restart otelcol.service

the service then keeps failing to restart, with the following output from journalctl

Stopped OpenTelemetry Collector.
-- Subject: A stop job for unit otelcol.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A stop job for unit otelcol.service has finished.
-- 
-- The job identifier is 5122 and the job result is done.
Jun 14 11:58:01 kushal-Nitro-AN515-31 systemd[1]: otelcol.service: Start request repeated too quickly.
Jun 14 11:58:01 kushal-Nitro-AN515-31 systemd[1]: otelcol.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit otelcol.service has entered the 'failed' state with result 'exit-code'.
Jun 14 11:58:01 kushal-Nitro-AN515-31 systemd[1]: Failed to start OpenTelemetry Collector.
-- Subject: A start job for unit otelcol.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit otelcol.service has finished with a failure

[kushal-ti]

is it better to just use otelcol as a normal process than to execute it as a systemd service ?

[fatsheep9146]

Yes, I suggest that you open two terminals, one for collector, one for demo app. And start the collector directly without systemd

[kushal-ti]

okay, so i tried this, and it didn't quite work.

i tried with this configuration

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

  opencensus:

  # Collect own metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: "otel-collector"
          scrape_interval: 10s
          static_configs:
            - targets: ["0.0.0.0:8888"]

processors:
  batch:

exporters:
  logging:
    logLevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [logging]

  extensions: [health_check, pprof, zpages]

and started it as you suggested, in another terminal window with

/usr/bin/otelcol --config /etc/otelcol/config.yaml

however, i don't see anything being printed to console.
Just to confirm the traces are being received in the first place, i tried to change the exporter to a file and it worked, however, the logging exporter doesn't seem to work (i also tried to use the logging exporter for exporting application logs, which also failed)

[fatsheep9146]

exporters:
  logging:
    logLevel: debug

should be

exporters:
  logging:
    loglevel: debug

logLevel is not right key. The right name is loglevel.

Try this way again. if it still does not work, can you share the stdout of collector to me?

[kushal-ti]

Hey, so sorry, i had forgotten to update here, but this worked perfectly, thanks alot

[fatsheep9146]

Please change the discussion into Answered. :)