From 5b05852110f7f2a19633627943882fb728b813fa Mon Sep 17 00:00:00 2001 From: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com> Date: Tue, 22 Mar 2022 08:59:37 -0400 Subject: [PATCH] Implement OTEP 0178: Mapping external data to AnyValue (#2385) OTEP 0178 [0] proposed this mapping. This PR merges the proposal into specification. The content of this PR is mostly copy/paste of the text of the OTEP minus the irrelevant sections such as "Alternates", etc. 0 - https://github.com/open-telemetry/oteps/blob/main/text/0178-mapping-to-otlp-anyvalue.md --- CHANGELOG.md | 5 + .../common/attribute-type-mapping.md | 255 ++++++++++++++++++ specification/common/common.md | 3 + 3 files changed, 263 insertions(+) create mode 100644 specification/common/attribute-type-mapping.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 72ecbf69dc1..0317a4d6239 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,11 @@ release. ### Telemetry Schemas +### Common + +- Add "Mapping Arbitrary Data to OTLP AnyValue" document. + ([#2385](https://github.com/open-telemetry/opentelemetry-specification/pull/2385)) + ## v1.9.0 (2021-02-10) ### Context diff --git a/specification/common/attribute-type-mapping.md b/specification/common/attribute-type-mapping.md new file mode 100644 index 00000000000..46a25c1504e --- /dev/null +++ b/specification/common/attribute-type-mapping.md @@ -0,0 +1,255 @@ +# Mapping Arbitrary Data to OTLP AnyValue + +**Status**: [Experimental](../document-status.md) + +
+Table of Contents + + + +- [Converting to AnyValue](#converting-to-anyvalue) + * [Primitive Values](#primitive-values) + + [Integer Values](#integer-values) + + [Enumerations](#enumerations) + + [Floating Point Values](#floating-point-values) + + [String Values](#string-values) + + [Byte Sequences](#byte-sequences) + * [Composite Values](#composite-values) + + [Array Values](#array-values) + + [Associative Arrays With Unique Keys](#associative-arrays-with-unique-keys) + + [Associative Arrays With Non-Unique Keys](#associative-arrays-with-non-unique-keys) + + [Sets](#sets) + * [Other Values](#other-values) + * [Empty Values](#empty-values) + + + +
+ +This document defines how to map (convert) arbitrary data (e.g. in-memory +objects) to OTLP's [AnyValue](https://github.com/open-telemetry/opentelemetry-proto/blob/cc4ed55c082cb75e084d40b4ddf3805eda099f97/opentelemetry/proto/common/v1/common.proto#L27). + +The mapping is needed when OpenTelemetry needs to convert a value produced outside +OpenTelemetry into a value that can be exported using an OTLP exporter, or otherwise be +converted to be used inside OpenTelemetry boundaries. Example use cases are the following: + +- In [Logging Library SDK](../logs/logging-library-sdk.md)s, to convert values received + from logging libraries into OpenTelemetry representation. +- In the Collector, to convert values received from various data sources into + [pdata](https://github.com/open-telemetry/opentelemetry-collector/blob/4998703dadd19fa91a88eabf7ccc68d728bee3fd/model/pdata/common.go#L84) + internal representation. + +## Converting to AnyValue + +[AnyValue](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L27) +is capable of representing primitive and structured data of certain types. + +Implementations that have source data in any form, such as in-memory objects +or data coming from other formats that needs to be converted to AnyValue SHOULD +follow the rules described below. + +### Primitive Values + +#### Integer Values + +Integer values which are within the range of 64 bit signed numbers +[-2^63..2^63-1] SHOULD be converted to AnyValue's +[int_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L33) +field. + +Integer values which are outside the range of 64 bit signed numbers SHOULD be +converted to AnyValue's +[string_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L31) +field using decimal representation. + +#### Enumerations + +Values, which belong to a limited enumerated set (e.g. a Java +[enum](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html)), SHOULD be +converted to AnyValue's +[string_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L31) +field with the value of the string set to the symbolic name of the enumeration. + +If it is not possible to obtain the symbolic name of the enumeration, the +implementation SHOULD map enumeration values to AnyValue's +[int_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L33) +field set to the enum's ordinal value, when such an ordinal number is naturally +obtainable. + +If it is also not possible to obtain the ordinal value, the enumeration SHOULD be +converted to AnyValue's +[bytes_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L37) +field in any manner that the implementation deems reasonable. + +#### Floating Point Values + +Floating point values which are within the range and precision of IEEE 754 +64-bit floating point numbers (including IEEE 32-bit floating point values) +SHOULD be converted to AnyValue's +[double_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L34) +field. + +Floating point values which are outside the range or precision of IEEE 754 +64-bit floating point numbers (e.g. IEEE 128-bit floating bit values) SHOULD be +converted to AnyValue's +[string_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L31) +field using decimal floating point representation. + +#### String Values + +String values which are valid UTF-8 sequences SHOULD be converted to +AnyValue's +[string_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L31) +field. + +String values which are not valid Unicode sequences SHOULD be converted to +AnyValue's +[bytes_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L37) +with the bytes representing the string in the original order and format of the +source string. + +#### Byte Sequences + +Byte sequences (e.g. Go's `[]byte` slice or raw byte content of a file) SHOULD +be converted to AnyValue's +[bytes_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L37) +field. + +### Composite Values + +#### Array Values + +Values that represent ordered sequences of other values (such as +[arrays](https://docs.oracle.com/javase/specs/jls/se7/html/jls-10.html), +[vectors](https://en.cppreference.com/w/cpp/container/vector), ordered +[lists](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists), +[slices](https://golang.org/ref/spec#Slice_types)) SHOULD be converted to +AnyValue's +[array_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L35) +field. String values and byte sequences are an exception from this rule (see +above). + +The rules described in this document should be applied recursively to each element +of the array. + +#### Associative Arrays With Unique Keys + +Values that represent associative arrays with unique keys (also often known +as maps, dictionaries or key-value stores) SHOULD be converted to AnyValue's +[kvlist_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L36) +field. + +If the keys of the source array are not strings, they MUST be converted to +strings by any means available, often via a toString() or stringify functions +available in programming languages. The conversion function MUST be chosen in a +way that ensures that the resulting string keys are unique in the target array. + +The value part of each element of the source array SHOULD be converted to +AnyValue recursively. + +For example a JSON object `{"a": 123, "b": "def"}` SHOULD be converted to + +``` +AnyValue{ + kvlist_value:KeyValueList{ + values:[ + KeyValue{key:"a",value:AnyValue{int_value:123}}, + KeyValue{key:"b",value:AnyValue{string_value:"def"}}, + ] + } +} +``` + +The rules described in this document should be applied recursively to each value +of the associative array. + +#### Associative Arrays With Non-Unique Keys + +Values that represent an associative arrays with non-unique keys where multiple values may be associated with the same key (also sometimes known +as multimaps, multidicts) SHOULD be converted to AnyValue's +[kvlist_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L36) +field. + +The resulting +[kvlist_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L36) +field MUST list each key only once and the value of each element of +[kvlist_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L36) +field MUST be an array represented using AnyValue's +[array_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L35) +field, each element of the +[array_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L35) +representing one value of source array associated with the given key. + +For example an associative array shown in the following table: + +|Key|Value| +|---|---| +|"abc"|123| +|"def"|"foo"| +|"def"|"bar"| + +SHOULD be converted to: + +``` +AnyValue{ + kvlist_value:KeyValueList{ + values:[ + KeyValue{ + key:"abc", + value:AnyValue{array_value:ArrayValue{values[ + AnyValue{int_value:123} + ]}} + }, + KeyValue{ + key:"def", + value:AnyValue{array_value:ArrayValue{values[ + AnyValue{string_value:"foo"}, + AnyValue{string_value:"bar"} + ]}} + }, + ] + } +} +``` + +The rules described in this document should be applied recursively to each value +of the associative array. + +#### Sets + +Unordered collections of unique values (such as +[Java Sets](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Set.html), +[C++ sets](https://en.cppreference.com/w/cpp/container/set), +[Python Sets](https://docs.python.org/3/tutorial/datastructures.html#sets)) SHOULD be +converted to AnyValue's +[array_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L35) +field, where each element of the set becomes an element of the array. + +The rules described in this document should be applied recursively to each value +of the set. + +### Other Values + +Any other values not listed above SHOULD be converted to AnyValue's +[string_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L31) +field if the source data can be serialized to a string (can be stringified) +using toString() or stringify functions available in programming languages. + +If the source data cannot be serialized to a string then the value SHOULD be +converted AnyValue's +[bytes_value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L37) +field by serializing it into a byte sequence by any means available. + +If the source data cannot be serialized neither to a string nor to a byte +sequence then it SHOULD by converted to an empty AnyValue. + +### Empty Values + +If the source data has no type associated with it and is empty, null, nil or +otherwise indicates absence of data it SHOULD be converted to an +[empty](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L29) +AnyValue, where all the fields are unset. + +Empty values which has a type associated with them (e.g. empty associative +array) SHOULD be converted using the corresponding rules defined for the types +above. diff --git a/specification/common/common.md b/specification/common/common.md index 774c2361302..6c3e80a527c 100644 --- a/specification/common/common.md +++ b/specification/common/common.md @@ -51,6 +51,9 @@ both containing an array of strings to represent a mapping See [Attribute Naming](attribute-naming.md) for naming guidelines. +See [this document](attribute-type-mapping.md) to find out how to map values obtained +outside OpenTelemetry into OpenTelemetry attribute values. + ### Attribute Limits Execution of erroneous code can result in unintended attributes. If there are no