ottl

OpenTelemetry Transformation Language

Status
Stability	alpha: traces, metrics, logs
Issues
Code Owners	@TylerHelmuth, @kentquirk, @bogdandrutu, @evan-bradley | Seeking more code owners!

The OpenTelemetry Transformation Language is a language for transforming open telemetry data based on the OpenTelemetry Collector Processing Exploration.

This package reads in OTTL statements and converts them to invokable functions/booleans based on the OTTL's grammar.

• Getting Started
• Examples
• Troubleshooting
• Resources

Getting Started

If you're looking to write OTTL statements for a component's configuration check out these resources.

See OTTL Functions for a list of functions available for use in the OTTL statements of most components.

OTTL Contexts define how you access the fields on a piece of telemetry. See the table to find the exact list of available fields:

Telemetry	OTTL Context
Resource	Resource
Instrumentation Scope	Instrumentation Scope
Span	Span
Span Event	SpanEvent
Metric	Metric
Datapoint	DataPoint
Log	Log

Component Creators

If you're looking to use OTTL in your component, check out the OTTL grammar.

Examples

These examples contain a SQL-like declarative language. Applied statements interact with only one signal, but statements can be declared across multiple signals. Functions used in examples are indicative of what could be useful.

Remove a forbidden attribute

traces:
  delete(attributes["http.request.header.authorization"])
metrics:
  delete(attributes["http.request.header.authorization"])
logs:
  delete(attributes["http.request.header.authorization"])

Remove all attributes except for some

traces:
  keep_keys(attributes, ["http.method", "http.status_code"])
metrics:
  keep_keys(attributes, ["http.method", "http.status_code"])
logs:
  keep_keys(attributes, ["http.method", "http.status_code"])

Reduce cardinality of an attribute

traces:
  replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}")

Reduce cardinality of a span name

traces:
  replace_match(name, "GET /user/*/list/*", "GET /user/{userId}/list/{listId}")

Reduce cardinality of any matching attribute

traces:
  replace_all_matches(attributes, "/user/*/list/*", "/user/{userId}/list/{listId}")

Decrease the size of the telemetry payload

traces:
  delete(resource.attributes["process.command_line"])
metrics:
  delete(resource.attributes["process.command_line"])
logs:
  delete(resource.attributes["process.command_line"])

Attach information from resource into telemetry

metrics:
  set(attributes["k8s_pod"], resource.attributes["k8s.pod.name"])

Decorate error spans with additional information

traces:
  set(attributes["whose_fault"], "theirs") where attributes["http.status"] == 400 or attributes["http.status"] == 404
  set(attributes["whose_fault"], "ours") where attributes["http.status"] == 500

Update a spans ID

logs:
  set(span_id, SpanID(0x0000000000000000))
traces:
  set(span_id, SpanID(0x0000000000000000))

Convert metric name to snake case

metrics:
  set(metric.name, ConvertCase(metric.name, "snake"))

Check if an attribute exists

traces:
  set(attributes["test-passed"], true) where attributes["target-attribute"] != nil

Troubleshooting

When using OTTL you can enable debug logging in the collector to print out useful information, such as the current Statement/Condition and the current TransformContext, to help you troubleshoot why a statement is not behaving as you expect. This feature is very verbose, but provides you an accurate view into how OTTL views the underlying data.

service:
  telemetry:
    logs:
      level: debug

2024-05-29T16:38:09.600-0600    debug   [email protected]/parser.go:265     initial TransformContext        {"kind": "processor", "name": "transform", "pipeline": "logs", "TransformContext": {"resource": {"attributes": {}, "dropped_attribute_count": 0}, "scope": {"attributes": {}, "dropped_attribute_count": 0, "name": "", "version": ""}, "log_record": {"attributes": {"log.file.name": "test.log"}, "body": "test", "dropped_attribute_count": 0, "flags": 0, "observed_time_unix_nano": 1717022289500721000, "severity_number": 0, "severity_text": "", "span_id": "", "time_unix_nano": 0, "trace_id": ""}, "cache": {}}}
2024-05-29T16:38:09.600-0600    debug   [email protected]/parser.go:268     TransformContext after statement execution      {"kind": "processor", "name": "transform", "pipeline": "logs", "statement": "set(resource.attributes[\"test\"], \"pass\")", "condition matched": true, "TransformContext": {"resource": {"attributes": {"test": "pass"}, "dropped_attribute_count": 0}, "scope": {"attributes": {}, "dropped_attribute_count": 0, "name": "", "version": ""}, "log_record": {"attributes": {"log.file.name": "test.log"}, "body": "test", "dropped_attribute_count": 0, "flags": 0, "observed_time_unix_nano": 1717022289500721000, "severity_number": 0, "severity_text": "", "span_id": "", "time_unix_nano": 0, "trace_id": ""}, "cache": {}}}
2024-05-29T16:38:09.600-0600    debug   [email protected]/parser.go:268     TransformContext after statement execution      {"kind": "processor", "name": "transform", "pipeline": "logs", "statement": "set(instrumentation_scope.attributes[\"test\"], [\"pass\"])", "condition matched": true, "TransformContext": {"resource": {"attributes": {"test": "pass"}, "dropped_attribute_count": 0}, "scope": {"attributes": {"test": ["pass"]}, "dropped_attribute_count": 0, "name": "", "version": ""}, "log_record": {"attributes": {"log.file.name": "test.log"}, "body": "test", "dropped_attribute_count": 0, "flags": 0, "observed_time_unix_nano": 1717022289500721000, "severity_number": 0, "severity_text": "", "span_id": "", "time_unix_nano": 0, "trace_id": ""}, "cache": {}}}
2024-05-29T16:38:09.601-0600    debug   [email protected]/parser.go:268     TransformContext after statement execution      {"kind": "processor", "name": "transform", "pipeline": "logs", "statement": "set(attributes[\"test\"], true)", "condition matched": true, "TransformContext": {"resource": {"attributes": {"test": "pass"}, "dropped_attribute_count": 0}, "scope": {"attributes": {"test": ["pass"]}, "dropped_attribute_count": 0, "name": "", "version": ""}, "log_record": {"attributes": {"log.file.name": "test.log", "test": true}, "body": "test", "dropped_attribute_count": 0, "flags": 0, "observed_time_unix_nano": 1717022289500721000, "severity_number": 0, "severity_text": "", "span_id": "", "time_unix_nano": 0, "trace_id": ""}, "cache": {}}}

Resources

These are previous conference presentations given about OTTL:

• OTTL Me Why Transforming Telemetry in the OpenTelemetry Collector Just Got Better
• Managing Observability Data at the Edge with the OpenTelemetry Collector and OTTL
• The OTTL Cookbook: A Collection of Solutions to Common Problems