[tracking] Jaeger v2 based on OpenTelemetry collector

[yurishkuro]

Proposal: https://docs.google.com/document/d/1s4_6VgAS7qAVp6iEm5KYvpiGw3h2Ja5T5HpKo29iv00/view

This replaces the previous attempt: #3500.

Work tracked in the project https://github.com/orgs/jaegertracing/projects/3

Roadmap

Proof of concept

• A single binary with memory storage and all-in-one functionality
  • Add jaeger-v2 single binary based on OTEL collector #4766
• Validate that the current storage configuration works for primary / archive storage.
• Default configuration solution (do not require config file for all-in-one)
  • feat(jaeger-v2): Create default config for all-in-one #4842
• Demonstrate Kafka exporter / receiver (ideally from OTEL), compatible with existing Jaeger Kafka writer
  • [Jaeger v2] Add Kafka exporter and receiver #4868
• Define v2 storage API that operates on OTLP batches, see if internal & remote APIs can be identical
  • Define internal version of OTLP proto types using Gogo optimizations #5052
  • Implement Storage API v2 that operates on OTLP batches #5079
• Benchmarks [Jaeger v2] Run benchmarks #4869

Feature parity

• Extend configuration of jaeger_storage_exporter to support all existing backends -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084
• Metrics & Traces -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084
• What to do with admin port / services -- [jaeger-v2] Document changes to admin port / services documentation#750
• Adaptive sampling processor -- project in Jaeger's LFX Mentorship - 2024 Term 01, March - May #5084

Prepare for Beta

• Add to integration tests
• Add to release pipeline
• Documentation

Prepare for GA

• Helm chart [Feature]: Support Jaeger-v2 Helm Chart helm-charts#534
• Operator [Feature]: Support Jaeger-v2 Operator jaeger-operator#2411
• Clarify https://github.com/jaegertracing/jaeger#version-compatibility-guarantees

Enhancements

• SPM built-in
• Integrate with OTEL collector builder ocb
• Print active configuration (unresolved: Print OTEL configuration #5223), Auto Generation of Component Configuration from metadata.yaml  open-telemetry/opentelemetry-collector-contrib#27003

[pavolloffay]

The V2 is as well affected by open-telemetry/opentelemetry-collector#8768, It can cause startup failure.

[yurishkuro]

The new jaeger binary is available as Docker image https://hub.docker.com/repository/docker/jaegertracing/jaeger/tags

[yurishkuro]

I wrote down some thoughts on what Jaeger Storage API v2 might look like. Interested in feedback.

[hdost]

I wrote down some thoughts on what Jaeger Storage API v2 might look like. Interested in feedback.

So one thing that I think might be worth thinking about as well is search complexity. One issue I know we had was making search efficient on large span storage.
Time range to limit is useful, but painful for users. Being able to search on attributes is a must. It might be good add criteria to compare about query patterns as well.

[tmeckel]

When I try to use the OLTPLogExporter with Jeager and GRPC,

details collased

```python get_logger_provider().add_log_record_processor( BatchLogRecordProcessor( exporter=OTLPLogExporter( endpoint="http://jaeger:4317", insecure=True, ), ) ) ```

I recieve the following error:

2024-02-27 13:41:34,233 ERROR [opentelemetry.exporter.otlp.proto.grpc.exporter] [exporter.py:306] - Failed to export logs to jaeger:4317, error code: StatusCode.UNIMPLEMENTED

An indication that https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/logs/v1/logs_service.proto is not implemented by Jaeger. Is this correct? Or did I configured Jaeger incorrectly, ref. to following Docker Compose snippet (ports 4317 and 4318 are used only internally using the network ipam)?

---
# This is a Docker Compose override file to add jeager tracing
version: "3.9"

services:
  jaeger:
    image: jaegertracing/all-in-one:${JAEGER_VERSION:-latest}
    ports:
      - "16686:16686"
    environment:
      - LOG_LEVEL=debug
      - COLLECTOR_OTLP_ENABLED=true
    networks:
      - ipam

  network-managemnent:
    depends_on:
      - jaeger

open-telemetry/opentelemetry-collector#6363

[yurishkuro]

@tmeckel Jaeger is not a logging backend, it only accepts traces.