Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

util/tracing: remove the jaeger exporter #111342

Merged
merged 1 commit into from
Oct 2, 2023
Merged

Conversation

knz
Copy link
Contributor

@knz knz commented Sep 27, 2023

Fixes #110632.
Epic: CRDB-28893
Many thanks to @maxnilz for their suggestions.

Release note (backward-incompatible change): The direct export of traces to Jaeger and the cluster setting trace.jaeger.agent have been removed. The direct export functionality had been obsoleted since 2022; it stopped working altogether some time in 2023 with the following error:

data does not fit within one UDP packet; size 65006, max 65000, spans NN

Since 2022, Jaeger supports ingestion of traces using OLTP; and CockroachDB has supported emitting traces using OLTP since v22.1. Operators and developers who want to inspect traces are thus invited to use the OLTP protocol instead. The corresponding cluster setting is trace.opentelemetry.collector.

For a successful deployment, an intermediate OLTP collector/forwarder should be configured. This docker-compose configuration is suitable:

otel-collector:
  image: otel/opentelemetry-collector-contrib
  container_name: otel-collector
  volumes:
    - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
  ports:
    - 1888:1888 # pprof extension
    - 8888:8888 # Prometheus metrics exposed by the collector
    - 8889:8889 # Prometheus exporter metrics
    - 13133:13133 # health_check extension
    - 4317:4317 # OTLP gRPC receiver
    - 4318:4318 # OTLP http receiver
    - 55679:55679 # zpages extension

jaeger:
  image: jaegertracing/all-in-one
  container_name: jaeger
  ports:
    - "16685:16685"
    - "16686:16686"
    - "14250:14250"
    - "14268:14268"
    - "14269:14269"
    - "6831:6831/udp"
  environment:
    - COLLECTOR_ZIPKIN_HTTP_PORT=9411
    - COLLECTOR_OTLP_ENABLED=true

Together with the following otel-collector configuration:

receivers:
  otlp: # the OTLP receiver the app is sending traces to
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/jaeger: # Jaeger supports OTLP directly
    endpoint: http://jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/dev:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

To use this, unset jaeger SET CLUSTER SETTING trace.jaeger.agent='' then set the OLTP collector using SET CLUSTER SETTING trace.opentelemetry.collector='localhost:4317'.

Release note: None

@knz knz requested review from a team and Santamaura and removed request for a team September 27, 2023 15:02
@blathers-crl
Copy link

blathers-crl bot commented Sep 27, 2023

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@knz knz requested a review from a team as a code owner September 27, 2023 15:03
Copy link
Contributor

@abarganier abarganier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: after conflicts fixed

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @knz and @Santamaura)


-- commits line 13 at r1:
nit: s/OLTP/OTLP


-- commits line 81 at r1:
nit: "Release note" already given up top?

Release note (backward-incompatible change): The direct export of
traces to Jaeger and the cluster setting `trace.jaeger.agent` have
been removed. The direct export functionality had been obsoleted since
2022; it stopped working altogether some time in 2023 with the
following error:
```
data does not fit within one UDP packet; size 65006, max 65000, spans NN
```

Since 2022, Jaeger supports ingestion of traces using OTLP; and
CockroachDB has supported emitting traces using OTLP since v22.1.
Operators and developers who want to inspect traces are thus invited
to use the OTLP protocol instead. The corresponding cluster setting is
`trace.opentelemetry.collector`.

For a successful deployment, an intermediate OTLP collector/forwarder
should be configured. This docker-compose configuration is suitable:

```yaml
otel-collector:
  image: otel/opentelemetry-collector-contrib
  container_name: otel-collector
  volumes:
    - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
  ports:
    - 1888:1888 # pprof extension
    - 8888:8888 # Prometheus metrics exposed by the collector
    - 8889:8889 # Prometheus exporter metrics
    - 13133:13133 # health_check extension
    - 4317:4317 # OTLP gRPC receiver
    - 4318:4318 # OTLP http receiver
    - 55679:55679 # zpages extension

jaeger:
  image: jaegertracing/all-in-one
  container_name: jaeger
  ports:
    - "16685:16685"
    - "16686:16686"
    - "14250:14250"
    - "14268:14268"
    - "14269:14269"
    - "6831:6831/udp"
  environment:
    - COLLECTOR_ZIPKIN_HTTP_PORT=9411
    - COLLECTOR_OTLP_ENABLED=true
```

Together with the following otel-collector configuration:
```yaml
receivers:
  otlp: # the OTLP receiver the app is sending traces to
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/jaeger: # Jaeger supports OTLP directly
    endpoint: http://jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/dev:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]
```

To use this, unset jaeger `SET CLUSTER SETTING trace.jaeger.agent=''`
then set the OTLP collector using `SET CLUSTER SETTING
trace.opentelemetry.collector='localhost:4317'`.
Copy link
Contributor Author

@knz knz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @abarganier and @Santamaura)


-- commits line 13 at r1:

Previously, abarganier (Alex Barganier) wrote…

nit: s/OLTP/OTLP

Done


-- commits line 81 at r1:

Previously, abarganier (Alex Barganier) wrote…

nit: "Release note" already given up top?

Removed

@knz knz force-pushed the 20230927-jaeger branch from 0b2a9af to 4b4dafa Compare October 2, 2023 16:17
@knz
Copy link
Contributor Author

knz commented Oct 2, 2023

TFYR!

bors r=abarganier

@craig
Copy link
Contributor

craig bot commented Oct 2, 2023

Build succeeded:

@craig craig bot merged commit 9addf20 into cockroachdb:master Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tracer: Jaeger tracer agent report fail
3 participants