How to create multiple tracers?

[maiermic]

The example in the Readme only shows how to create/initialize the GlobalTracer. How to create another (local) tracer (e.g. for SQL spans)?

At least, I'd like to get spans in jaeger shown in a different color if they do not belong to my backend, but to the database. See screenshot from Jaeger documentation:

Are multiple tracers the wrong approach for this?

[sergeyklay]

Never used more than one tracer. In the provided screenshot, different spans are marked with the same color (and not just the database). Could you give an example/screenshot of what you are trying to change?

[maiermic]

I'm using Doctrine in my PHP backend. I configure a SQLLogger in Doctrine to add spans for the SQL operations

TracingSQLLogger.php

<?php

use Doctrine\DBAL\Logging\SQLLogger;
use OpenTracing\GlobalTracer;

class TracingSQLLogger implements SQLLogger
{
  private \OpenTracing\Scope $scope;
  
  public function startQuery($sql, ?array $params = null, ?array $types = null)
  {
    $tracer = GlobalTracer::get();
    $this->scope = $tracer->startActiveSpan('sql');
  }

  public function stopQuery()
  {
    $this->scope->close();
  }
}

Among others, I defined a middleware to add a span for a request

TracingMiddleware.php

<?php

use OpenTracing\GlobalTracer;
use \Psr\Http\Message\ServerRequestInterface as Request;
use \Psr\Http\Message\ResponseInterface as Response;

class TracingMiddleware
{
  /**
   * @throws \Throwable
   */
  public function __invoke(Request $request, Response $response, callable $next): Response
  {
    $tracer = GlobalTracer::get();
    $scope = $tracer->startActiveSpan('request', []);
    $span = $scope->getSpan();
    $span->setTag("http.method", $request->getMethod());
    $span->setTag("http.url", $request->getUri());
    try {
      /** @var Response $result */
      $result = $next($request, $response);
    } catch (\Throwable $e) {
      $scope->close();
      throw $e;
    }
    $span->setTag("http.status_code", $result->getStatusCode());
    $scope->close();
    return $result;
  }
}

I add more spans that are backend related (e.g. main and request.authentication) in other places of the backend. However, since all use the same tracer (i.e. GlobalTracer), all spans have the same color in Jaeger UI

I'd like to have different colors (and names) for different services. I trace the SQL queries in my backend, but they actually belong to the service sql and not to the service backend. I can create two configurations with different service names

$backendConfig = new \Jaeger\Config($config, 'backend');
$sqlConfig = new \Jaeger\Config($config, 'sql');

but how do I create two different tracers? The tracer with $backendConfig should be used globally. So I can call

$backendConfig->initializeTracer();

The tracer with $sqlConfig should only be used in TracingSQLLogger. But how do I create the tracer? Will

$sqlConfig->initializeTracer();

overwrite the global tracer returned by GlobalTracer::get()? In that case, I could not use my backend tracer anymore 😞

Are multiple tracers the wrong approach for this?

How do you add spans of your SQL/database queries to the trace?

[sergeyklay]

Got it. Well, I'm not sure this is configurable by jaeger-client-php. I'm even not sure this being supported by the protocol. As far as I know Jaeger UI is responsible for color definition.

[pekhota]

Your problem is more related to the jaeger ui rather than jaeger client. There is an old issue there jaegertracing/jaeger-ui#38. You can get some details from it. Also jaeger has some config options https://www.jaegertracing.io/docs/1.26/frontend-ui/ for their ui. Maybe it would be helpful for you.

Overall I don't recommend you to use multiple tracers because 1 instance of tracer means 1 application you are going to trace. Having more than 1 tracer means that you will break this abstraction. Maybe you need add some tag to database spans and use ui search for that.

But, if you insist here is the solution:

$tracers = [];

for ($i = 0; $i < 5; $i++) {
    $config = new Config(
        [
            'sampler' => [
                'type' => Jaeger\SAMPLER_TYPE_CONST,
                'param' => true,
            ],
            'logging' => true,
            'dispatch_mode' => Config::JAEGER_OVER_BINARY_UDP,
        ],
        sprintf('your-app-name%d', $i),
        new StdErrLogger()
    );

    $reporter = (new JaegerReporterFactory($config))->createReporter();
    $sampler = new ConstSampler(true);

    $tracer = $config->createTracer($reporter, $sampler);
    $tracers[] = $tracer;
}


foreach ($tracers as $tracer) {
    $scope1 = $tracer->startActiveSpan('TestSpan1', []);

    for ($j = 0; $j < 4; $j++) {
        $scope2 = $tracer->startActiveSpan(sprintf("Child %d; level 1", $j));
        for ($i = 0; $i < 520; $i++) {
            $scope3 = $tracer->startActiveSpan(sprintf("Child %d level 2", $i));
            $scope3->close();
        }
        $scope2->close();
    }

    $scope1->close();

    $tracer->flush();
}

[maiermic]

There is an old issue there jaegertracing/jaeger-ui#38.

That's a different issue about using different colors for the same service name:

The trace view currently colors spans by the service name only, so calls between different instances of the same service are colored the same

I'd like to trace the SQL service (different service name than backend). AFAIK this can not be done in the SQL service (server) itself and has to be done by the SQL client (i.e. my PHP backend). Therefore, I somehow have to change the service name.

As far as I understand, this is what the Hot R.O.D. example (written in Go) does. It changes the service name (as you can see in this article)

I'm not sure, how this is done in the Go client and how it is supposed to be done in the PHP client. The Jaeger\Config takes a service name as argument, but maybe this is not the correct way to change the service of a SQL span.

Is it a peer service? It seems to me that the tag peer.service is set in services/customer/database.go. If I set this tag in the PHP client

$span->setTag("peer.service", "mysql");

the tag is strangely not displayed in the Jaeger UI. Why? Is this tag maybe not supported by the PHP client? Or am I missing something else?

Do you even trace SQL service?

[maiermic]

The Hot R.O.D. example runs multiple services, but the customer service has two tracer. The first is created with service name customer

server := customer.NewServer(
	net.JoinHostPort("0.0.0.0", strconv.Itoa(customerPort)),
	tracing.Init("customer", metricsFactory, logger),
	metricsFactory,
	logger,
)

In customer.NewServer the second tracer with service name mysql is created

func NewServer(hostPort string, tracer opentracing.Tracer, metricsFactory metrics.Factory, logger log.Factory) *Server {
	return &Server{
		hostPort: hostPort,
		tracer:   tracer,
		logger:   logger,
		database: newDatabase(
			tracing.Init("mysql", metricsFactory, logger),
			logger.With(zap.String("component", "mysql")),
		),
	}
}

So I guess, the correct way are multiple tracers to trace SQL service from your client.

[maiermic]

Thanks, for your help. It is working 😎

<?php
require_once __DIR__.'/lib/vendor/autoload.php';

use Jaeger\Config;
use Jaeger\ReporterFactory\JaegerReporterFactory;
use Jaeger\Sampler\ConstSampler;
use OpenTracing\GlobalTracer;

$sharedJaegerConfiguration = [
  'sampler' => [
    'type' => Jaeger\SAMPLER_TYPE_CONST,
    'param' => true,
  ],
  'logging' => true,
  "tags" => [],
  "local_agent" => [
    "reporting_host" => "jaeger",
  ],
  'dispatch_mode' => Config::JAEGER_OVER_BINARY_UDP,
];
$globalConfig = new \Jaeger\Config(
  $sharedJaegerConfiguration,
  'backend'
);
$globalConfig->initializeTracer();
$config = new \Jaeger\Config(
  $sharedJaegerConfiguration,
  'mysql'
);
$globalTracer = GlobalTracer::get();

$scope = $globalTracer->startActiveSpan('BackendSpan', []);
$backendSpan = $scope->getSpan();

$reporter = (new JaegerReporterFactory($config))->createReporter();
$sampler = new ConstSampler(true);
$sqlTracer = $config->createTracer($reporter, $sampler);
$nestedSpanScope = $sqlTracer->startActiveSpan("MySQLSpan", [
  'child_of' => $backendSpan,
]);
sleep(1);
$nestedSpanScope->close();
$sqlTracer->flush();

sleep(1);
$scope->close();
$globalTracer->flush();