From 86b65ac180ab99575f8dce56f6e7ec4f4219675d Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 23 Feb 2023 10:59:27 -0800 Subject: [PATCH 01/30] Configuration proposal This OTEP is the result of the Configuration working group. See https://github.com/open-telemetry/opentelemetry-specification/issues/2920 for additional details. cc @MrAlias @jack-berg Signed-off-by: Alex Boten --- text/0225-configuration.md | 411 +++++++++++++++++++++++++++++++++++++ 1 file changed, 411 insertions(+) create mode 100644 text/0225-configuration.md diff --git a/text/0225-configuration.md b/text/0225-configuration.md new file mode 100644 index 000000000..822966327 --- /dev/null +++ b/text/0225-configuration.md @@ -0,0 +1,411 @@ +# OpenTelemetry Configuration + +A new configuration interface is proposed here in the form of a configuration model, which can be expressed as a file, and validated through a published schema. + +## Motivation + +OpenTelemetry specifies code that can operate in a variety of ways based on the end-user’s desired mode of operation. This requires a configuration interface be provided to the user so they are able to communicate this information. Currently, OpenTelemetry specifies this interface in the form of the API exposed by the SDKs and environment variables. This environment variable interface is limited in the structure of information it can communicate and the primitives it can support. + +### Environment Variable Interface Limitations + +The environment variable interface suffers from the following identified limitations: + +* **Flat**. Structured data is only allowed by using higher-level naming or data encoding schemes. Examples of configuration limited by lack of structured configuration include: + * Configuring multiple span processors, periodic metric readers, or log record processors. + * Configuring views. + * Configuring arguments for parent based sampler (sampler parent is remote and sampled vs. not sampled, sampler when parent is local and sampled vs. not sampled). +* **Runtime dependent**. Different systems expose this interface differently (Linux, BSD, Windows). This usually means unique instructions are required to properly interact with the configuration interface on different systems. +* **Limited values**. Many systems only allow string values to be used, but OpenTelemetry specifies many configuration values other than this type. For example, OTEL_RESOURCE_ATTRIBUTES specifies a list of key value pairs to be used as resource attributes, but there is no way to specify array values, or indicate that the value should be interpreted as non-string type. +* **Limited validation**. Validation can only be performed by the receiver, there is no meta-configuration language to validate input. +* **Difficult to extend**. It’s difficult to anticipate the requirements of configuring custom extension components (processors, exporters, samplers, etc), and likely not practical to represent them in a flat structure. As a result, the environment variable interface is limited to components explicitly mentioned in the specification. + +## Explanation + +Using a configuration model or configuration file, users can configure all options currently available via environment variables. + +### Goals + +* The configuration must be language implementation agnostic. It must not contain structure or statements that only can be interpreted in a subset of languages. This does not preclude the possibility that the configuration can have specific extensions included for a subset of languages, but it does mean that the configuration must be interpretable by all implementation languages. +* Broadly supported format. Ideally, the information encoded in the file can be decoded using native tools for all OpenTelemetry implementation languages. However, it must be possible for languages that do not natively support an encoding format to write their own parsers. +* The configuration format must support structured data. At the minimum arrays and associative arrays. +* The format must support at least null, boolean, string, double precision floating point (IEEE 754-1985), or signed 64 bit integer value types. +* Custom span processors, exporters, samplers, or other user defined extension components can be configured using this format. +* Configure SDK, but also configure instrumentation. +* Must offer stability guarantees while supporting evolution +* The structure of the configuration can be validated via a schema. +* Support environment variable substitution to give users the option to avoid storing secrets in these files. + +## Internal details + +The schema for OpenTelemetry configuration is to be published in a repository to allow language implementations to leverage that definition to automatically generate code and/or validate end-user configuration. This will ensure that all implementations provide a consistent experience for any version of the schema they support. An example of such a proposed schema is available [here](https://github.com/MrAlias/otel-schema/tree/main/json_schema/schema). + +The working group proposes the use of [JSON Schema](https://json-schema.org/) as the language to define the schema. It provides: + +* support for client-side validation +* code generation +* broad support across languages + +In order to provide a minimal API surface area, implementations *MUST* support the following methods. + +### Configure(config) + +An API called `Configure` receives a configuration object. This method then applies the configuration object's details to the SDK. This method specifically applies the configuration object to allow for multiple configuration format providers to be supported in the future. This OTEP describes two such providers in a file and data structure formats below, but remote file formats *MAY* be implemented in the future. + +### ParseAndValidateConfigurationFromFile(filepath, format) -> config + +An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. YAML *SHOULD* be supported, but JSON, which is a strict subset of YAML, *MAY* be supported if a language ecosystem has good reason. The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: + +* file doesn't exist or is invalid +* configuration parsed is invalid + +#### Python ParseAndValidateConfigurationFromFile example + +```python + +filename = "./config.yaml" + +try: + cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filename) +except Exception as e: + print(e) + +filename = "./config.json" + +try: + cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filename, format="json") +except Exception as e: + raise e + +``` + +#### Go ParseAndValidateConfigurationFromFile example + +```go + +filename := "./config.yaml" +cfg, err := otel.ParseAndValidateConfigurationFromFile(filename) +if err != nil { + return err +} + +filename := "./config.json" +cfg, err := otel.ParseAndValidateConfigurationFromFile(filename, otelconfig.WithFormat("json")) +if err != nil { + return err +} + +``` + +Implementations *MUST* allow users to specify an environment variable to set the configuration file. This gives flexibility to end users of implementations that do not support command line arguments. Some possible names for this variable: + +* `OTEL_CONFIGURATION_FILE` +* `OTEL_CONFIG_FILE` +* `OTEL_CONF` + +### Configuration model + +To allow SDKs and instrumentation libraries to accept configuration without having to implement the parsing logic, a `Configuration` model *MAY* be provided by implementations. This object: + +* has already been parsed from a file or data structure +* is structurally valid (errors may yet occur when SDK or instrumentation interprets the object) + +### Additional interface: ParseAndValidateConfiguration + +Each language implementation supporting OpenTelemetry *MAY* support parsing a data structure instead of a file to produce a model. This allows implementations to provide a configuration interface without the expectation on users to parse a configuration file. The following demonstrates how Python and Go may provide a configuration interface to accomplish this: + +#### Python ParseAndValidateConfiguration example + +```python +opentelemetry.ParseAndValidateConfiguration( + { + "scheme_version": "0.0.1", + "sdk": { + "resource": { + "attributes": { + "service.name": "unknown_service", + } + }, + "propagators": ["tracecontext", "baggage", "b3multi"], + "tracer_provider": { + "exporters": { + "zipkin": { + "endpoint": "http://localhost:9411/api/v2/spans", + "timeout": 10000, + }, + "otlp": {}, + }, + "span_processors": [ + { + "name": "batch", + "args": { + "schedule_delay": 5000, + "export_timeout": 30000, + "max_queue_size": 2048, + "max_export_batch_size": 512, + "exporter": "zipkin", + }, + } + ], + }, + ... + } + }) +``` + +### Go ParseAndValidateConfiguration example + +```go +type config map[string]interface{} // added for convenience + +otel.ParseAndValidateConfiguration(config{ + "sdk": config{ + "resource": config{ + "attributes": config{ + "service.name": "unknown_service", + }, + }, + "propagators": []string{"tracecontext", "baggage", "b3multi"}, + "tracer_provider": config{ + "exporters": config{ + "zipkin": config{ + "endpoint": "http://localhost:9411/api/v2/spans", + "timeout": 10000, + }, + "otlp": config{}, + }, + "span_processors": []config{ + { + "name": "batch", + "args": config{ + "schedule_delay": 5000, + "export_timeout": 30000, + "max_queue_size": 2048, + "max_export_batch_size": 512, + "exporter": "zipkin", + }, + }, + }, + }, + ... + }, + }, +) +``` + +### Configuration file + +The following demonstrates an example of a configuration file format (full example [here](https://github.com/MrAlias/otel-schema/blob/main/config.yaml)): + +```yaml +# include version specification in configuration files to help with parsing and schema evolution. +scheme_version: 0.1 +sdk: + # Disable the SDK for all signals. + # + # Boolean value. If "true", a no-op SDK implementation will be used for all telemetry + # signals. Any other value or absence of the variable will have no effect and the SDK + # will remain enabled. This setting has no effect on propagators configured through + # the OTEL_PROPAGATORS variable. + # + # Environment variable: OTEL_SDK_DISABLED + disabled: false + # Configure resource attributes and resource detection for all signals. + resource: + # Key-value pairs to be used as resource attributes. + # + # Environment variable: OTEL_RESOURCE_ATTRIBUTES + attributes: + # Sets the value of the `service.name` resource attribute + # + # Environment variable: OTEL_SERVICE_NAME + service.name: !!str "unknown_service" + # Configure context propagators. Each propagator has a name and args used to configure it. None of the propagators here have configurable options so args is not demonstrated. + # + # Environment variable: OTEL_PROPAGATORS + propagators: [tracecontext, baggage] + # Configure the tracer provider. + tracer_provider: + # Span exporters. Each exporter key refers to the type of the exporter. Values configure the exporter. Exporters must be associated with a span processor. + exporters: + # Configure the zipkin exporter. + zipkin: + # Sets the endpoint. + # + # Environment variable: OTEL_EXPORTER_ZIPKIN_ENDPOINT + endpoint: http://localhost:9411/api/v2/spans + # Sets the max time to wait for each export. + # + # Environment variable: OTEL_EXPORTER_ZIPKIN_TIMEOUT + timeout: 10000 + # List of span processors. Each span processor has a name and args used to configure it. + span_processors: + # Add a batch span processor. + # + # Environment variable: OTEL_BSP_*, OTEL_TRACES_EXPORTER + - name: batch + # Configure the batch span processor. + args: + # Sets the delay interval between two consecutive exports. + # + # Environment variable: OTEL_BSP_SCHEDULE_DELAY + schedule_delay: 5000 + # Sets the maximum allowed time to export data. + # + # Environment variable: OTEL_BSP_EXPORT_TIMEOUT + export_timeout: 30000 + # Sets the maximum queue size. + # + # Environment variable: OTEL_BSP_MAX_QUEUE_SIZE + max_queue_size: 2048 + # Sets the maximum batch size. + # + # Environment variable: OTEL_BSP_MAX_EXPORT_BATCH_SIZE + max_export_batch_size: 512 + # Sets the exporter. Exporter must refer to a key in sdk.tracer_provider.exporters. + # + # Environment variable: OTEL_TRACES_EXPORTER + exporter: zipkin + # Configure the meter provider. + ... +``` + +### Environment variable substitution + +Configuration files *MUST* support environment variable expansion. While this accommodates the scenario in which a configuration file needs to reference sensitive data and is not able to be stored securely, environment variable expansion is not limited to sensitive data. + +The syntax for environment variable expansion mirrors the [collector](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables). + +For example, given an environment where `API_KEY=1234`, the configuration file contents: + +```yaml +scheme_version: 0.1 +sdk: + tracer_provider: + exporters: + otlp: + endpoint: https://example.host:4317/v1/traces + headers: + api-key: ${env:API_KEY} +``` + +Result in the following after substitution: + +```yaml +scheme_version: 0.1 +sdk: + tracer_provider: + exporters: + otlp: + endpoint: https://example.host:4317/v1/traces + headers: + api-key: 1234 +``` + +Implementations *MUST* perform environment variable substitution before validating and parsing configuration file contents. + +### Version guarantees & backwards compatibility + +Each version of the configuration schema carries a major and minor version. Configurations specify the major and minor version they adhere to. Before reaching 1.0, each minor version change is equivalent to major version change. That is, there are no guarantees about compatibility and all changes are permitted. As of 1.0, we provide the following stability guarantees: + +* For major version: No guarantees. +* For minor versions: + * Property keys will not change. Although the addition of optional properties is permitted and the removal of properties is permitted, this *MUST* not be abused to rename property keys. + * Property value types will not change, except that integers MAY become floating points. + * No additional required properties. + +Allowable changes: + +* For major versions: All changes are permitted. +* For minor versions: + * Addition of optional properties. + * Required property keys may become optional. + * Removal of properties, provided that the property key is not reused in the future. + +SDKs validating configuration *MUST* fail when they encounter a configuration with an unsupported version. Generally, this means fail when encountering a major version which is not recognized. An SDK might choose to maintain a library of validators / parsers for each major version, and use the configuration version to select and use the correct instance. Differences in minor versions (except pre-1.0 minor versions) SHOULD be acceptable, with the caveat that allowable property additions and removals MAY result in configuration that is different than excepted. + +## Trade-offs and mitigations + +### Additional method to configure OpenTelemetry + +If the implementation suggested in this OTEP goes ahead, users will be presented with another mechanism for configuring OpenTelemetry. This may cause confusion for users who are new to the project. It may be possible to mitigate the confusion by providing users with best practices and documentation. + +### Many ways to configure may result in users not knowing what is configured + +As there are multiple mechanisms for configuration, it's possible that the active configuration isn't what was expected. This could happen today, and one way it could be mitigated would be by providing a mechanism to list the active OpenTelemetry configuration. + +### Errors or difficulty in configuration files + +Configuration files provide an opportunity for misconfiguration. A way to mitigate this would be to provide clear messaging and fail quickly when misconfiguration occurs. + +## Prior art and alternatives + +The working group looked to the OpenTelemetry Collector and OpenTelemetry Operator for inspiration and guidance. + +### Alternative schema languages + +In choosing to recommend JSON schema, the working group looked at the following options: + +* [Cue](https://cuelang.org/) - A promising simpler language to define a schema, the working group decided against CUE because: + * Tooling available for validating CUE files in languages outside of Go were limited. + * Familiarity and learning curve would create problems for both users and contributors of OpenTelemetry. +* [Protobuf](https://developers.google.com/protocol-buffers) - With protobuf already used heavily in OpenTelemetry, the format was worth investigating as an option to define the schema. The working group decided against Protobuf because: + * Validation errors are the result of serlization errors which can be difficult to interpret. + * Limitations in the schema definition language result in poor ergonomics if type safety is to be retained. + +## Open questions + +### How to handle environment variable / file config overlap? + +How does file configuration interact with environment variable configuration when both are present? + +* Solution 1: Ignore environment configuration when file configuration is present. Log a warning to the user indicating that multiple configuration modes were detected, but use the file configuration as the source of truth. +* Solution 2: Superimpose environment configuration on top of file configuration when both are present. One problem with this is that environment variable configuration doesn’t map to file configuration in an intuitive way. For example, OTEL_TRACES_EXPORTER defines a list of span exporters to be paired with a batch span processor configured by the OTEL_BSP_* variables. What do we do if the file config already contains one or more processors with an exporter specified in OTEL_TRACES_EXPORTER? Essentially, do we merge or append the environment variable configuration? + +### How to handle no-code vs programmatic configuration? + +How should the SDK be configured when both no-code configuration (either environment variable or file config) and programmatic configuration are present? NOTE: this question exists today with only the environment variable interface available. + +* Solution 1: Make it clear that interpretation of the environment shouldn’t be built into components. Instead, SDKs should have a component that explicitly interprets the environment and returns a configured instance of the SDK. This is how the java SDK works today and it nicely separates concerns. + +## Future possibilities + +### Additional configuration providers + +Although the initial proposal for configuration supports only describes in-code and file representations, it's possible additional sources (remote, opamp, ...) for configuration will be desirable. The implementation of the configuration model and components should be extensible to allow for this. + +### Integration with auto-instrumentation + +The configuration model could be integrated to work with the existing auto-instrumentation tooling in each language implementation. + +#### Java + +The Java implementation provides a JAR that supports configuring various parameters via system properties. This implementation could leverage a configuration file by supporting its configuration a system property: + +```bash +java -javaagent:path/to/opentelemetry-javaagent.jar \ + -Dotel.config=./config.yaml + -jar myapp.jar +``` + +#### Python + +The Python implementation has a command available that allows users to leverage auto-instrumentation. The `opentelemetry-instrument` command could use a `--config` flag to pass in a config file: + +```bash +# install the instrumentation package +pip install opentelemetry-instrumentation +# use a --config parameter to pass in the configuration file +# NOTE: this parameter does not currently exist and would need to be added +opentelemetry-instrument --config ./config.yaml ./python/app.py +``` + +#### OpAmp + +The configuration may be used in the future in conjunction with the OpAmp protocol to make remote configuration of SDKs available as a feature supported by OpenTelemetry. + +## Related Spec issues address + +* [https://github.com/open-telemetry/opentelemetry-specification/issues/1773](https://github.com/open-telemetry/opentelemetry-specification/issues/1773) +* [https://github.com/open-telemetry/opentelemetry-specification/issues/2857](https://github.com/open-telemetry/opentelemetry-specification/issues/2857) +* [https://github.com/open-telemetry/opentelemetry-specification/issues/2746](https://github.com/open-telemetry/opentelemetry-specification/issues/2746) +* [https://github.com/open-telemetry/opentelemetry-specification/issues/2860](https://github.com/open-telemetry/opentelemetry-specification/issues/2860) From 70ba4e029890e6fe787d03f6a5ac531221027173 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 23 Feb 2023 14:36:33 -0800 Subject: [PATCH 02/30] update based on feedback Signed-off-by: Alex Boten --- text/0225-configuration.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 822966327..6743ea487 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -18,6 +18,7 @@ The environment variable interface suffers from the following identified limitat * **Limited values**. Many systems only allow string values to be used, but OpenTelemetry specifies many configuration values other than this type. For example, OTEL_RESOURCE_ATTRIBUTES specifies a list of key value pairs to be used as resource attributes, but there is no way to specify array values, or indicate that the value should be interpreted as non-string type. * **Limited validation**. Validation can only be performed by the receiver, there is no meta-configuration language to validate input. * **Difficult to extend**. It’s difficult to anticipate the requirements of configuring custom extension components (processors, exporters, samplers, etc), and likely not practical to represent them in a flat structure. As a result, the environment variable interface is limited to components explicitly mentioned in the specification. +* **Lacks versioning**. The lack of versioning support for environment variables prevents evolution over time. ## Explanation @@ -28,7 +29,7 @@ Using a configuration model or configuration file, users can configure all optio * The configuration must be language implementation agnostic. It must not contain structure or statements that only can be interpreted in a subset of languages. This does not preclude the possibility that the configuration can have specific extensions included for a subset of languages, but it does mean that the configuration must be interpretable by all implementation languages. * Broadly supported format. Ideally, the information encoded in the file can be decoded using native tools for all OpenTelemetry implementation languages. However, it must be possible for languages that do not natively support an encoding format to write their own parsers. * The configuration format must support structured data. At the minimum arrays and associative arrays. -* The format must support at least null, boolean, string, double precision floating point (IEEE 754-1985), or signed 64 bit integer value types. +* The format must support at least null, boolean, string, double precision floating point (IEEE 754-1985), and signed 64 bit integer value types. * Custom span processors, exporters, samplers, or other user defined extension components can be configured using this format. * Configure SDK, but also configure instrumentation. * Must offer stability guarantees while supporting evolution @@ -96,11 +97,9 @@ if err != nil { ``` -Implementations *MUST* allow users to specify an environment variable to set the configuration file. This gives flexibility to end users of implementations that do not support command line arguments. Some possible names for this variable: +Implementations *MUST* allow users to specify an environment variable to set the configuration file. This gives flexibility to end users of implementations that do not support command line arguments. The proposed name for this variable: -* `OTEL_CONFIGURATION_FILE` * `OTEL_CONFIG_FILE` -* `OTEL_CONF` ### Configuration model From 4221c3e8e20a4526e5381167ac5f3fcd202262d8 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 24 Feb 2023 08:31:03 -0800 Subject: [PATCH 03/30] rename scheme_version to file_format Signed-off-by: Alex Boten --- text/0225-configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 6743ea487..dda2350d2 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -117,7 +117,7 @@ Each language implementation supporting OpenTelemetry *MAY* support parsing a da ```python opentelemetry.ParseAndValidateConfiguration( { - "scheme_version": "0.0.1", + "file_format": "0.0.1", "sdk": { "resource": { "attributes": { @@ -197,7 +197,7 @@ The following demonstrates an example of a configuration file format (full examp ```yaml # include version specification in configuration files to help with parsing and schema evolution. -scheme_version: 0.1 +file_format: 0.1 sdk: # Disable the SDK for all signals. # @@ -277,7 +277,7 @@ The syntax for environment variable expansion mirrors the [collector](https://op For example, given an environment where `API_KEY=1234`, the configuration file contents: ```yaml -scheme_version: 0.1 +file_format: 0.1 sdk: tracer_provider: exporters: @@ -290,7 +290,7 @@ sdk: Result in the following after substitution: ```yaml -scheme_version: 0.1 +file_format: 0.1 sdk: tracer_provider: exporters: From 3d5276f88195f6cae2e1962ab07e18feb6939c06 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 24 Feb 2023 13:17:51 -0800 Subject: [PATCH 04/30] Update text/0225-configuration.md Co-authored-by: Reiley Yang --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index dda2350d2..0063c9ab9 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -32,7 +32,7 @@ Using a configuration model or configuration file, users can configure all optio * The format must support at least null, boolean, string, double precision floating point (IEEE 754-1985), and signed 64 bit integer value types. * Custom span processors, exporters, samplers, or other user defined extension components can be configured using this format. * Configure SDK, but also configure instrumentation. -* Must offer stability guarantees while supporting evolution +* Must offer stability guarantees while supporting evolution. * The structure of the configuration can be validated via a schema. * Support environment variable substitution to give users the option to avoid storing secrets in these files. From ac57e8999d2450216718c18a04e95e9ea852072f Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 24 Feb 2023 13:26:14 -0800 Subject: [PATCH 05/30] add reasoning for json support Signed-off-by: Alex Boten --- text/0225-configuration.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 0063c9ab9..2579de125 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -54,7 +54,9 @@ An API called `Configure` receives a configuration object. This method then appl ### ParseAndValidateConfigurationFromFile(filepath, format) -> config -An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. YAML *SHOULD* be supported, but JSON, which is a strict subset of YAML, *MAY* be supported if a language ecosystem has good reason. The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: +An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. YAML *SHOULD* be supported, but JSON, which is a strict subset of YAML, *MAY* be supported if a language ecosystem has good reason. For example a language that has built-in support for JSON running in an environment that is sensitive to additional dependencies *MAY* only be able to support JSON configuration. + +The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: * file doesn't exist or is invalid * configuration parsed is invalid From 78fdfcaf80615c994e459b828eda6ccc5d8a852d Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 24 Feb 2023 13:28:00 -0800 Subject: [PATCH 06/30] Update text/0225-configuration.md Co-authored-by: Reiley Yang --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 2579de125..6fd877b70 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -65,7 +65,7 @@ The method returns a `Configuration` model that has been validated. This API *MA ```python -filename = "./config.yaml" +filepath = "./config.yaml" try: cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filename) From d06d6f66efd0edb30b0169bf60bc24284b18a714 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 24 Feb 2023 15:32:29 -0800 Subject: [PATCH 07/30] remove free-form dictionary --- text/0225-configuration.md | 83 -------------------------------------- 1 file changed, 83 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 6fd877b70..a4bf34fdb 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -110,89 +110,6 @@ To allow SDKs and instrumentation libraries to accept configuration without havi * has already been parsed from a file or data structure * is structurally valid (errors may yet occur when SDK or instrumentation interprets the object) -### Additional interface: ParseAndValidateConfiguration - -Each language implementation supporting OpenTelemetry *MAY* support parsing a data structure instead of a file to produce a model. This allows implementations to provide a configuration interface without the expectation on users to parse a configuration file. The following demonstrates how Python and Go may provide a configuration interface to accomplish this: - -#### Python ParseAndValidateConfiguration example - -```python -opentelemetry.ParseAndValidateConfiguration( - { - "file_format": "0.0.1", - "sdk": { - "resource": { - "attributes": { - "service.name": "unknown_service", - } - }, - "propagators": ["tracecontext", "baggage", "b3multi"], - "tracer_provider": { - "exporters": { - "zipkin": { - "endpoint": "http://localhost:9411/api/v2/spans", - "timeout": 10000, - }, - "otlp": {}, - }, - "span_processors": [ - { - "name": "batch", - "args": { - "schedule_delay": 5000, - "export_timeout": 30000, - "max_queue_size": 2048, - "max_export_batch_size": 512, - "exporter": "zipkin", - }, - } - ], - }, - ... - } - }) -``` - -### Go ParseAndValidateConfiguration example - -```go -type config map[string]interface{} // added for convenience - -otel.ParseAndValidateConfiguration(config{ - "sdk": config{ - "resource": config{ - "attributes": config{ - "service.name": "unknown_service", - }, - }, - "propagators": []string{"tracecontext", "baggage", "b3multi"}, - "tracer_provider": config{ - "exporters": config{ - "zipkin": config{ - "endpoint": "http://localhost:9411/api/v2/spans", - "timeout": 10000, - }, - "otlp": config{}, - }, - "span_processors": []config{ - { - "name": "batch", - "args": config{ - "schedule_delay": 5000, - "export_timeout": 30000, - "max_queue_size": 2048, - "max_export_batch_size": 512, - "exporter": "zipkin", - }, - }, - }, - }, - ... - }, - }, -) -``` - ### Configuration file The following demonstrates an example of a configuration file format (full example [here](https://github.com/MrAlias/otel-schema/blob/main/config.yaml)): From fb39f8b9adae70d777df46ccf939b817df412af8 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Feb 2023 11:31:38 -0800 Subject: [PATCH 08/30] Update text/0225-configuration.md Co-authored-by: Reiley Yang --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index a4bf34fdb..24589f687 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -68,7 +68,7 @@ The method returns a `Configuration` model that has been validated. This API *MA filepath = "./config.yaml" try: - cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filename) + cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filepath) except Exception as e: print(e) From 782b35cb5c4c7b260279053c3c5d85847973b323 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Feb 2023 12:15:51 -0800 Subject: [PATCH 09/30] update env variable expansion to reflect that the current collector syntax is a starting point here. Signed-off-by: Alex Boten --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 24589f687..e2909456e 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -191,7 +191,7 @@ sdk: Configuration files *MUST* support environment variable expansion. While this accommodates the scenario in which a configuration file needs to reference sensitive data and is not able to be stored securely, environment variable expansion is not limited to sensitive data. -The syntax for environment variable expansion mirrors the [collector](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables). +As a starting point for development, the syntax for environment variable expansion *MAY* mirror the [collector](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables). For example, given an environment where `API_KEY=1234`, the configuration file contents: From de34c6fe0f3da568b160363801b012b1c4356ad1 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Feb 2023 12:21:06 -0800 Subject: [PATCH 10/30] update should to must --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index e2909456e..439804839 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -239,7 +239,7 @@ Allowable changes: * Required property keys may become optional. * Removal of properties, provided that the property key is not reused in the future. -SDKs validating configuration *MUST* fail when they encounter a configuration with an unsupported version. Generally, this means fail when encountering a major version which is not recognized. An SDK might choose to maintain a library of validators / parsers for each major version, and use the configuration version to select and use the correct instance. Differences in minor versions (except pre-1.0 minor versions) SHOULD be acceptable, with the caveat that allowable property additions and removals MAY result in configuration that is different than excepted. +SDKs validating configuration *MUST* fail when they encounter a configuration with an unsupported version. Generally, this means fail when encountering a major version which is not recognized. An SDK might choose to maintain a library of validators / parsers for each major version, and use the configuration version to select and use the correct instance. Differences in minor versions (except pre-1.0 minor versions) *MUST* be acceptable, with the caveat that allowable property additions and removals MAY result in configuration that is different than excepted. ## Trade-offs and mitigations From 98d31b98d890edacc51d8be4f74486b6b03aaa4f Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Feb 2023 13:06:35 -0800 Subject: [PATCH 11/30] move env + file conflict to env variable section Signed-off-by: Alex Boten --- text/0225-configuration.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 439804839..92ffbbf1f 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -221,6 +221,10 @@ sdk: Implementations *MUST* perform environment variable substitution before validating and parsing configuration file contents. +#### Handling environment variable & file config overlap + +When both configuration file and environment variables are present, the implementation *MUST* ignore environment variables in preference of the configuration file. The support for environment variable substitution in the configuration file gives users a mechanism for migrating away from environment variables in favour of configuration files. + ### Version guarantees & backwards compatibility Each version of the configuration schema carries a major and minor version. Configurations specify the major and minor version they adhere to. Before reaching 1.0, each minor version change is equivalent to major version change. That is, there are no guarantees about compatibility and all changes are permitted. As of 1.0, we provide the following stability guarantees: @@ -272,13 +276,6 @@ In choosing to recommend JSON schema, the working group looked at the following ## Open questions -### How to handle environment variable / file config overlap? - -How does file configuration interact with environment variable configuration when both are present? - -* Solution 1: Ignore environment configuration when file configuration is present. Log a warning to the user indicating that multiple configuration modes were detected, but use the file configuration as the source of truth. -* Solution 2: Superimpose environment configuration on top of file configuration when both are present. One problem with this is that environment variable configuration doesn’t map to file configuration in an intuitive way. For example, OTEL_TRACES_EXPORTER defines a list of span exporters to be paired with a batch span processor configured by the OTEL_BSP_* variables. What do we do if the file config already contains one or more processors with an exporter specified in OTEL_TRACES_EXPORTER? Essentially, do we merge or append the environment variable configuration? - ### How to handle no-code vs programmatic configuration? How should the SDK be configured when both no-code configuration (either environment variable or file config) and programmatic configuration are present? NOTE: this question exists today with only the environment variable interface available. From 968a5d236b6b321966cd76098d949f054d1f736e Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Feb 2023 13:21:13 -0800 Subject: [PATCH 12/30] add out of scope section under goals Signed-off-by: Alex Boten --- text/0225-configuration.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 92ffbbf1f..c64ec65fe 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -36,6 +36,10 @@ Using a configuration model or configuration file, users can configure all optio * The structure of the configuration can be validated via a schema. * Support environment variable substitution to give users the option to avoid storing secrets in these files. +#### Out of scope + +* Embedding additional configuration files within a configuration file through an expression. Additional configuration providers MAY choose to support this use-case in the future. + ## Internal details The schema for OpenTelemetry configuration is to be published in a repository to allow language implementations to leverage that definition to automatically generate code and/or validate end-user configuration. This will ensure that all implementations provide a consistent experience for any version of the schema they support. An example of such a proposed schema is available [here](https://github.com/MrAlias/otel-schema/tree/main/json_schema/schema). From 253ba05bd7a36ac7920b10c04a2630ba4bc2bcb2 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 2 Mar 2023 11:43:44 -0800 Subject: [PATCH 13/30] add note about consistent mapping --- text/0225-configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index c64ec65fe..855ffbe2b 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -191,6 +191,8 @@ sdk: ... ``` +Note that there is no consistent mapping between environment variable names and the keys in the configuration file. + ### Environment variable substitution Configuration files *MUST* support environment variable expansion. While this accommodates the scenario in which a configuration file needs to reference sensitive data and is not able to be stored securely, environment variable expansion is not limited to sensitive data. From b0b0eff8d8600d17dd714e05a7b4978b6395cde7 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 2 Mar 2023 11:47:14 -0800 Subject: [PATCH 14/30] adding example of a custom processor in the config Signed-off-by: Alex Boten --- text/0225-configuration.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 855ffbe2b..8ddee75c9 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -187,6 +187,12 @@ sdk: # # Environment variable: OTEL_TRACES_EXPORTER exporter: zipkin + # custom processor + - name: my-custom-processor + args: + foo: bar + baz: qux + # Configure the meter provider. ... ``` From 4f9f55acbedea435febbb3e927084ebf738253bd Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 2 Mar 2023 12:15:21 -0800 Subject: [PATCH 15/30] add note about using file extension to determine format of file Signed-off-by: Alex Boten --- text/0225-configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 8ddee75c9..c1bc6d1f8 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -107,6 +107,8 @@ Implementations *MUST* allow users to specify an environment variable to set the * `OTEL_CONFIG_FILE` +The format for the configuration file will be detected using the file extension of this variable. + ### Configuration model To allow SDKs and instrumentation libraries to accept configuration without having to implement the parsing logic, a `Configuration` model *MAY* be provided by implementations. This object: From cec1e03ceab3d9ad8dbeefbcdb28ca14d56b3296 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Fri, 3 Mar 2023 10:47:25 -0800 Subject: [PATCH 16/30] Update text/0225-configuration.md Co-authored-by: Daniel Dyla --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index c1bc6d1f8..a405306d7 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -58,7 +58,7 @@ An API called `Configure` receives a configuration object. This method then appl ### ParseAndValidateConfigurationFromFile(filepath, format) -> config -An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. YAML *SHOULD* be supported, but JSON, which is a strict subset of YAML, *MAY* be supported if a language ecosystem has good reason. For example a language that has built-in support for JSON running in an environment that is sensitive to additional dependencies *MAY* only be able to support JSON configuration. +An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: From ba7764aff5f7c53db7c3f1c33598999832416417 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 6 Mar 2023 12:32:06 -0800 Subject: [PATCH 17/30] generalize parse method Signed-off-by: Alex Boten --- text/0225-configuration.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index a405306d7..a58cc888d 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -54,49 +54,50 @@ In order to provide a minimal API surface area, implementations *MUST* support t ### Configure(config) -An API called `Configure` receives a configuration object. This method then applies the configuration object's details to the SDK. This method specifically applies the configuration object to allow for multiple configuration format providers to be supported in the future. This OTEP describes two such providers in a file and data structure formats below, but remote file formats *MAY* be implemented in the future. +An API called `Configure` receives a configuration object. This method applies the configuration object's details to the SDK. This method specifically applies the configuration object to allow for multiple configuration format providers to be supported in the future. This OTEP describes two such providers in a file and data structure formats below, but remote file formats *MAY* be implemented in the future. -### ParseAndValidateConfigurationFromFile(filepath, format) -> config +### Parse(file) -> config -An API called `ParseAndValidateConfigurationFromFile` receives a string parameter indicating the file path containing the configuration to be parsed. An optional format parameter may be provided to indicate the format that this configuration uses. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. +An API called `Parse` receives a file object. The method loads the contents of the file, parses it, and validates that the configuration against the schema. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: * file doesn't exist or is invalid * configuration parsed is invalid -#### Python ParseAndValidateConfigurationFromFile example +#### Python Parse example ```python filepath = "./config.yaml" + try: - cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filepath) + cfg = opentelemetry.Parse(filepath) except Exception as e: print(e) -filename = "./config.json" +filepath = "./config.json" try: - cfg = opentelemetry.ParseAndValidateConfigurationFromFile(filename, format="json") + cfg = opentelemetry.Parse(filepath) except Exception as e: raise e ``` -#### Go ParseAndValidateConfigurationFromFile example +#### Go Parse example ```go -filename := "./config.yaml" -cfg, err := otel.ParseAndValidateConfigurationFromFile(filename) +filepath := "./config.yaml" +cfg, err := otel.Parse(filepath) if err != nil { return err } -filename := "./config.json" -cfg, err := otel.ParseAndValidateConfigurationFromFile(filename, otelconfig.WithFormat("json")) +filepath := "./config.json" +cfg, err := otel.Parse(filepath) if err != nil { return err } From a668552c6f9e2a326d937bb237748e6416885774 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 7 Mar 2023 13:51:22 -0800 Subject: [PATCH 18/30] Update text/0225-configuration.md Co-authored-by: jack-berg <34418638+jack-berg@users.noreply.github.com> --- text/0225-configuration.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index a58cc888d..887df7e92 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -245,20 +245,14 @@ When both configuration file and environment variables are present, the implemen Each version of the configuration schema carries a major and minor version. Configurations specify the major and minor version they adhere to. Before reaching 1.0, each minor version change is equivalent to major version change. That is, there are no guarantees about compatibility and all changes are permitted. As of 1.0, we provide the following stability guarantees: * For major version: No guarantees. -* For minor versions: - * Property keys will not change. Although the addition of optional properties is permitted and the removal of properties is permitted, this *MUST* not be abused to rename property keys. - * Property value types will not change, except that integers MAY become floating points. - * No additional required properties. +* For minor versions: TBD Allowable changes: * For major versions: All changes are permitted. -* For minor versions: - * Addition of optional properties. - * Required property keys may become optional. - * Removal of properties, provided that the property key is not reused in the future. +* For minor versions: TBD -SDKs validating configuration *MUST* fail when they encounter a configuration with an unsupported version. Generally, this means fail when encountering a major version which is not recognized. An SDK might choose to maintain a library of validators / parsers for each major version, and use the configuration version to select and use the correct instance. Differences in minor versions (except pre-1.0 minor versions) *MUST* be acceptable, with the caveat that allowable property additions and removals MAY result in configuration that is different than excepted. +SDKs validating configuration *MUST* fail when they encounter a configuration with an unsupported version. Generally, this means fail when encountering a major version which is not recognized. An SDK might choose to maintain a library of validators / parsers for each major version, and use the configuration version to select and use the correct instance. Differences in minor versions (except pre-1.0 minor versions) *MUST* be acceptable. ## Trade-offs and mitigations From c3e12d29a119cdbf92152208711d1c874e03a68a Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 16 Mar 2023 08:10:07 -0700 Subject: [PATCH 19/30] Update text/0225-configuration.md --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 887df7e92..96bec5357 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -63,7 +63,7 @@ An API called `Parse` receives a file object. The method loads the contents of t The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: * file doesn't exist or is invalid -* configuration parsed is invalid +* configuration parsed is invalid according to schema #### Python Parse example From 8407f94dd2f88d0317bbb74c48b3e241a2198089 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 20 Mar 2023 10:34:31 -0700 Subject: [PATCH 20/30] Update text/0225-configuration.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Robert Pająk --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 96bec5357..925e30102 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -29,7 +29,7 @@ Using a configuration model or configuration file, users can configure all optio * The configuration must be language implementation agnostic. It must not contain structure or statements that only can be interpreted in a subset of languages. This does not preclude the possibility that the configuration can have specific extensions included for a subset of languages, but it does mean that the configuration must be interpretable by all implementation languages. * Broadly supported format. Ideally, the information encoded in the file can be decoded using native tools for all OpenTelemetry implementation languages. However, it must be possible for languages that do not natively support an encoding format to write their own parsers. * The configuration format must support structured data. At the minimum arrays and associative arrays. -* The format must support at least null, boolean, string, double precision floating point (IEEE 754-1985), and signed 64 bit integer value types. +* The format must support at least boolean, string, double precision floating point (IEEE 754-1985), and signed 64 bit integer value types. * Custom span processors, exporters, samplers, or other user defined extension components can be configured using this format. * Configure SDK, but also configure instrumentation. * Must offer stability guarantees while supporting evolution. From 7d65b00545bc31379b8bdbe6870a12995957e39d Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Mon, 20 Mar 2023 16:43:33 -0500 Subject: [PATCH 21/30] Configure can be extended to update SDK components --- text/0225-configuration.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 925e30102..83109ac41 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -50,17 +50,13 @@ The working group proposes the use of [JSON Schema](https://json-schema.org/) as * code generation * broad support across languages -In order to provide a minimal API surface area, implementations *MUST* support the following methods. - -### Configure(config) - -An API called `Configure` receives a configuration object. This method applies the configuration object's details to the SDK. This method specifically applies the configuration object to allow for multiple configuration format providers to be supported in the future. This OTEP describes two such providers in a file and data structure formats below, but remote file formats *MAY* be implemented in the future. +In order to provide a minimal API surface area, implementations *MUST* support the following: ### Parse(file) -> config An API called `Parse` receives a file object. The method loads the contents of the file, parses it, and validates that the configuration against the schema. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. -The method returns a `Configuration` model that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: +The method returns a [Configuration model](#configuration-model) that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: * file doesn't exist or is invalid * configuration parsed is invalid according to schema @@ -110,6 +106,22 @@ Implementations *MUST* allow users to specify an environment variable to set the The format for the configuration file will be detected using the file extension of this variable. +### Configurer + +`Configurer` interprets a [Configuration model](#configuration-model) and produces configured SDK components. + +Multiple `Configurer`s can be [created](#createconfig---configurer) with different configurations. It is the caller's responsibility to ensure the [resulting SDK components](#get-tracerprovider-meterprovider-loggerprovider) are correctly wired into the application and instrumentation. + +`Configurer` **MAY** be extended in the future with functionality to apply an updated configuration model to the resulting SDK components. + +#### Create(config) -> Configurer + +Create a `Configurer` from a [configuration model](#configuration-model). + +#### Get TracerProvider, MeterProvider, LoggerProvider + +Interpret the [configuration model](#configuration-model) and return SDK TracerProvider, MeterProvider, LoggerProvider which strictly reflect the configuration object's details and ignores the [opentelemetry environment variable configuration scheme](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md). + ### Configuration model To allow SDKs and instrumentation libraries to accept configuration without having to implement the parsing logic, a `Configuration` model *MAY* be provided by implementations. This object: From f2e579f6f5f8eff1bf62f7d8b81aa2002279ef30 Mon Sep 17 00:00:00 2001 From: Tyler Yahn Date: Tue, 21 Mar 2023 10:26:46 -0700 Subject: [PATCH 22/30] Add exact conf design to open questions --- text/0225-configuration.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 925e30102..3c1d16d8b 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -291,6 +291,26 @@ How should the SDK be configured when both no-code configuration (either environ * Solution 1: Make it clear that interpretation of the environment shouldn’t be built into components. Instead, SDKs should have a component that explicitly interprets the environment and returns a configured instance of the SDK. This is how the java SDK works today and it nicely separates concerns. +### What is the exact configuration file format to use? + +Included in this OTEP is an example configuration file format. +This included format was settled on so the configuration file schemas being proposed here could all be evaluated as viable options. +It acted as proof-of-concept that _a_ configuration file format existed that could describe needed OTel configuration and be described by a schema. +However, the configuration file format presented here is not meant as the final, nor optimal, design for use by OpenTelemetry. + +What that final design will be is left to discussion when this OTEP is implemented in the OpenTelemetry specification. +It is explicitly something this OTEP is not intended to resolved. + +The following are existing questions that will need to be resolved in the final design: + +1. Should the trace exporter be at the same level as the span processors? +2. Should the sampler config be separate from the sampler? +3. Is the `sdk` key appropriate? Should alternate configuration live under its own key but the SDKs configuration be at the top level? +4. Should the `disabled` key be renamed as `enabled`? + +This list is not intended as comprehensive. +There are likely more questions related to the final design that will be discussed when implemented in the OpenTelemetry specification. + ## Future possibilities ### Additional configuration providers From 87a51659e7144984f0b097583c4dbc6518df30e3 Mon Sep 17 00:00:00 2001 From: jack-berg <34418638+jack-berg@users.noreply.github.com> Date: Tue, 21 Mar 2023 13:10:51 -0500 Subject: [PATCH 23/30] Update text/0225-configuration.md --- text/0225-configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index ad0f51812..9ae5411da 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -248,6 +248,8 @@ sdk: Implementations *MUST* perform environment variable substitution before validating and parsing configuration file contents. +If a configuration file references an environment variable which is undefined, implementations *MUST* return an error or raise an exception. + #### Handling environment variable & file config overlap When both configuration file and environment variables are present, the implementation *MUST* ignore environment variables in preference of the configuration file. The support for environment variable substitution in the configuration file gives users a mechanism for migrating away from environment variables in favour of configuration files. From 85204bb3c08229026906473c4d822e219156f6dc Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 21 Mar 2023 14:31:15 -0700 Subject: [PATCH 24/30] fix linting Signed-off-by: Alex Boten --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 9ae5411da..f43ba9de2 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -248,7 +248,7 @@ sdk: Implementations *MUST* perform environment variable substitution before validating and parsing configuration file contents. -If a configuration file references an environment variable which is undefined, implementations *MUST* return an error or raise an exception. +If a configuration file references an environment variable which is undefined, implementations *MUST* return an error or raise an exception. #### Handling environment variable & file config overlap From 287cdbe9ef7614779e9c879fe6e5fe03eb5b552c Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Tue, 21 Mar 2023 15:52:58 -0700 Subject: [PATCH 25/30] remove difficule to extend Signed-off-by: Alex Boten --- text/0225-configuration.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index f43ba9de2..a7bfcf789 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -17,7 +17,6 @@ The environment variable interface suffers from the following identified limitat * **Runtime dependent**. Different systems expose this interface differently (Linux, BSD, Windows). This usually means unique instructions are required to properly interact with the configuration interface on different systems. * **Limited values**. Many systems only allow string values to be used, but OpenTelemetry specifies many configuration values other than this type. For example, OTEL_RESOURCE_ATTRIBUTES specifies a list of key value pairs to be used as resource attributes, but there is no way to specify array values, or indicate that the value should be interpreted as non-string type. * **Limited validation**. Validation can only be performed by the receiver, there is no meta-configuration language to validate input. -* **Difficult to extend**. It’s difficult to anticipate the requirements of configuring custom extension components (processors, exporters, samplers, etc), and likely not practical to represent them in a flat structure. As a result, the environment variable interface is limited to components explicitly mentioned in the specification. * **Lacks versioning**. The lack of versioning support for environment variables prevents evolution over time. ## Explanation From 19444556d823cfedd2643337f7f12633e959830e Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 22 Mar 2023 11:15:01 -0700 Subject: [PATCH 26/30] capture the need to decide on env var + config file conflicts Signed-off-by: Alex Boten --- text/0225-configuration.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index a7bfcf789..11421a543 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -251,7 +251,14 @@ If a configuration file references an environment variable which is undefined, i #### Handling environment variable & file config overlap -When both configuration file and environment variables are present, the implementation *MUST* ignore environment variables in preference of the configuration file. The support for environment variable substitution in the configuration file gives users a mechanism for migrating away from environment variables in favour of configuration files. +The behaviour when both configuration file and environment variables are present will be decided in the final design. Here are four options that should be considered: + +1. implementations ignore environment variables in preference of the configuration file +2. implementations give preference to the environment variables in preference over the configuration file +3. an exception arises causing the application to fail to start +4. the behaviour is left unspecified + +The support for environment variable substitution in the configuration file gives users a mechanism for migrating away from environment variables in favour of configuration files. ### Version guarantees & backwards compatibility From 82772d8cdb0b91d8f50152e04399936bb2c63cfe Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Wed, 22 Mar 2023 11:16:02 -0700 Subject: [PATCH 27/30] correct typo Signed-off-by: Alex Boten --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 11421a543..e9e284e4b 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -254,7 +254,7 @@ If a configuration file references an environment variable which is undefined, i The behaviour when both configuration file and environment variables are present will be decided in the final design. Here are four options that should be considered: 1. implementations ignore environment variables in preference of the configuration file -2. implementations give preference to the environment variables in preference over the configuration file +2. implementations give preference to the environment variables over the configuration file 3. an exception arises causing the application to fail to start 4. the behaviour is left unspecified From c59973d904ecbc39cf0963b1cbf487a1f21f31fb Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Thu, 23 Mar 2023 07:52:31 -0700 Subject: [PATCH 28/30] Update text/0225-configuration.md Co-authored-by: Tyler Benson --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index e9e284e4b..048c4d567 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -347,7 +347,7 @@ The Java implementation provides a JAR that supports configuring various paramet ```bash java -javaagent:path/to/opentelemetry-javaagent.jar \ - -Dotel.config=./config.yaml + -Dotel.config.file=./config.yaml -jar myapp.jar ``` From 49d897a71c2724b0110262460b98b4fb0f58bd8e Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Mar 2023 10:36:25 -0700 Subject: [PATCH 29/30] Update text/0225-configuration.md Co-authored-by: Armin Ruech --- text/0225-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 048c4d567..417f616b3 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -53,7 +53,7 @@ In order to provide a minimal API surface area, implementations *MUST* support t ### Parse(file) -> config -An API called `Parse` receives a file object. The method loads the contents of the file, parses it, and validates that the configuration against the schema. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. +An API called `Parse` receives a file object. The method loads the contents of the file, parses it, and validates that the configuration against the schema. At least one of JSON or YAML MUST be supported. If either format can be supported without additional dependencies, that format SHOULD be preferred. If neither or both formats are supported natively, YAML should be the preferred choice. If YAML is not supported due to dependency concerns, there MAY be a way for a user to explicitly enable it by installing their own dependency. The method returns a [Configuration model](#configuration-model) that has been validated. This API *MAY* return an error or raise an exception, whichever is idiomatic to the implementation for the following reasons: From a894597d183bebfeba9540a125ecce48c4b03295 Mon Sep 17 00:00:00 2001 From: Alex Boten Date: Mon, 27 Mar 2023 12:52:02 -0700 Subject: [PATCH 30/30] add config/schema examples to assets dir Signed-off-by: Alex Boten --- text/0225-configuration.md | 4 +- text/assets/0225-config.yaml | 420 ++++++++++++++++++++++ text/assets/0225-schema.json | 656 +++++++++++++++++++++++++++++++++++ 3 files changed, 1078 insertions(+), 2 deletions(-) create mode 100644 text/assets/0225-config.yaml create mode 100644 text/assets/0225-schema.json diff --git a/text/0225-configuration.md b/text/0225-configuration.md index 417f616b3..69d38c0e7 100644 --- a/text/0225-configuration.md +++ b/text/0225-configuration.md @@ -41,7 +41,7 @@ Using a configuration model or configuration file, users can configure all optio ## Internal details -The schema for OpenTelemetry configuration is to be published in a repository to allow language implementations to leverage that definition to automatically generate code and/or validate end-user configuration. This will ensure that all implementations provide a consistent experience for any version of the schema they support. An example of such a proposed schema is available [here](https://github.com/MrAlias/otel-schema/tree/main/json_schema/schema). +The schema for OpenTelemetry configuration is to be published in a repository to allow language implementations to leverage that definition to automatically generate code and/or validate end-user configuration. This will ensure that all implementations provide a consistent experience for any version of the schema they support. An example of such a proposed schema is available [here](./assets/0225-schema.json). The working group proposes the use of [JSON Schema](https://json-schema.org/) as the language to define the schema. It provides: @@ -130,7 +130,7 @@ To allow SDKs and instrumentation libraries to accept configuration without havi ### Configuration file -The following demonstrates an example of a configuration file format (full example [here](https://github.com/MrAlias/otel-schema/blob/main/config.yaml)): +The following demonstrates an example of a configuration file format (full example [here](./assets/0225-config.yaml)): ```yaml # include version specification in configuration files to help with parsing and schema evolution. diff --git a/text/assets/0225-config.yaml b/text/assets/0225-config.yaml new file mode 100644 index 000000000..febfd3ee8 --- /dev/null +++ b/text/assets/0225-config.yaml @@ -0,0 +1,420 @@ +# include version specification in configuration files to help with parsing and schema evolution. +scheme_version: 0.1 +sdk: + # Disable the SDK for all signals. + # + # Boolean value. If "true", a no-op SDK implementation will be used for all telemetry + # signals. Any other value or absence of the variable will have no effect and the SDK + # will remain enabled. This setting has no effect on propagators configured through + # the OTEL_PROPAGATORS variable. + # + # Environment variable: OTEL_SDK_DISABLED + disabled: false + # Configure resource attributes and resource detection for all signals. + resource: + # Key-value pairs to be used as resource attributes. + # + # Environment variable: OTEL_RESOURCE_ATTRIBUTES + attributes: + # Sets the value of the `service.name` resource attribute + # + # Environment variable: OTEL_SERVICE_NAME + service.name: !!str "unknown_service" + # Configure context propagators. Each propagator has a name used to configure it. + # + # Environment variable: OTEL_PROPAGATORS + propagators: [tracecontext, baggage, b3multi] + # Configure general attribute limits. See also sdk.tracer_provider.span_limits, sdk.logger_provider.log_record_limits. + attribute_limits: + # Set the max attribute value size. + # + # Environment variable: OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT + attribute_value_length_limit: 4096 + # Set the max attribute count. + # + # Environment variable: OTEL_ATTRIBUTE_COUNT_LIMIT + attribute_count_limit: 128 + # Configure the tracer provider. + tracer_provider: + # Span exporters. Each exporter key refers to the type of the exporter. Values configure the exporter. Exporters must be associated with a span processor. + exporters: + # Configure the otlp exporter. + otlp: + # Sets the protocol. + # + # Environment variable: OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL + protocol: http/protobuf + # Sets the endpoint. + # + # Environment variable: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT + endpoint: http://localhost:4318/v1/traces + # Sets the certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE + certificate: /app/cert.pem + # Sets the mTLS private client key. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_KEY, OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY + client_key: /app/cert.pem + # Sets the mTLS client certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE + client_certificate: /app/cert.pem + # Sets the headers. + # + # Environment variable: OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS + headers: + api-key: 1234 + # Sets the compression. + # + # Environment variable: OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSION + compression: gzip + # Sets the max time to wait for each export. + # + # Environment variable: OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_TRACES_TIMEOUT + timeout: 10000 + # Configure the zipkin exporter. + zipkin: + # Sets the endpoint. + # + # Environment variable: OTEL_EXPORTER_ZIPKIN_ENDPOINT + endpoint: http://localhost:9411/api/v2/spans + # Sets the max time to wait for each export. + # + # Environment variable: OTEL_EXPORTER_ZIPKIN_TIMEOUT + timeout: 10000 + # Configure the jaeger exporter. + jaeger: + # Sets the protocol. + # + # Environment variable: OTEL_EXPORTER_JAEGER_PROTOCOL + protocol: http/thrift.binary + # Sets the endpoint. Applicable when protocol is http/thrift.binary or grpc. + # + # Environment variable: OTEL_EXPORTER_JAEGER_ENDPOINT + endpoint: http://localhost:14268/api/traces + # Sets the max time to wait for each export. Applicable when protocol is http/thrift.binary or grpc. + # + # Environment variable: OTEL_EXPORTER_JAEGER_TIMEOUT + timeout: 10000 + # Sets the username for HTTP basic authentication. Applicable when protocol is http/thrift.binary or grpc. + # + # Environment variable: OTEL_EXPORTER_JAEGER_USER + user: user + # Sets the password for HTTP basic authentication. Applicable when protocol is http/thrift.binary or grpc. + # + # Environment variable: OTEL_EXPORTER_JAEGER_PASSWORD + password: password + # Sets the hostname of the Jaeger agent. Applicable when protocol is udp/thrift.compact or udp/thrift.binary. + # + # Environment variable: OTEL_EXPORTER_JAEGER_AGENT_PORT + agent_host: localhost + # Sets the port of the Jaeger agent. Applicable when protocol is udp/thrift.compact or udp/thrift.binary. + # + # Environment variable: OTEL_EXPORTER_JAEGER_AGENT_HOST + agent_port: 6832 + # List of span processors. Each span processor has a type and args used to configure it. + span_processors: + # Add a batch span processor. + # + # Environment variable: OTEL_BSP_*, OTEL_TRACES_EXPORTER + - type: batch + # Configure the batch span processor. + args: + # Sets the delay interval between two consecutive exports. + # + # Environment variable: OTEL_BSP_SCHEDULE_DELAY + schedule_delay: 5000 + # Sets the maximum allowed time to export data. + # + # Environment variable: OTEL_BSP_EXPORT_TIMEOUT + export_timeout: 30000 + # Sets the maximum queue size. + # + # Environment variable: OTEL_BSP_MAX_QUEUE_SIZE + max_queue_size: 2048 + # Sets the maximum batch size. + # + # Environment variable: OTEL_BSP_MAX_EXPORT_BATCH_SIZE + max_export_batch_size: 512 + # Sets the exporter. Exporter must refer to a key in sdk.tracer_provider.exporters. + # + # Environment variable: OTEL_TRACES_EXPORTER + exporter: otlp + # Add a batch span processor configured with zipkin exporter. For full description of options see sdk.tracer_provider.span_processors[0]. + - type: batch + args: + exporter: zipkin + # Add a batch span processor configured with jaeger exporter. For full description of options see sdk.tracer_provider.span_processors[0]. + - type: batch + args: + exporter: jaeger + # Configure the span limits. See also sdk.attribute_limits. + span_limits: + # Set the max span attribute value size. Overrides sdk.attribute_limits.attribute_value_length_limit. + # + # Environment variable: OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT + attribute_value_length_limit: 4096 + # Set the max span attribute count. Overrides sdk.attribute_limits.attribute_count_limit. + # + # Environment variable: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT + attribute_count_limit: 128 + # Set the max span event count. + # + # Environment variable: OTEL_SPAN_EVENT_COUNT_LIMIT + event_count_limit: 128 + # Set the max span link count. + # + # Environment variable: OTEL_SPAN_LINK_COUNT_LIMIT + link_count_limit: 128 + # Set the max attributes per span event. + # + # Environment variable: OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT + event_attribute_count_limit: 128 + # Set the max attributes per span link. + # + # Environment variable: OTEL_LINK_ATTRIBUTE_COUNT_LIMIT + link_attribute_count_limit: 128 + # Configuration for samplers. Each key refers to the type of sampler. Values configure the sampler. One key must be referenced in sdk.tracer_provider.sampler. + sampler_config: + # Configure the always_on sampler. + # + # Environment variable: OTEL_TRACES_SAMPLER=always_on + always_on: + # Configure the always_off sampler. + # + # Environment variable: OTEL_TRACES_SAMPLER=always_off + always_off: + # Configure the trace_id_ratio_based sampler. + # + # Environment variable: OTEL_TRACES_SAMPLER=traceidratio + trace_id_ratio_based: + # Set the sampling ratio. + # + # Environment variable: OTEL_TRACES_SAMPLER=traceidratio, OTEL_TRACES_SAMPLER_ARG=0.0001 + ratio: 0.0001 + # Configure the parent_based sampler. + # + # Environment variable: OTEL_TRACES_SAMPLER=parentbased_* + parent_based: + # Set root sampler. Must refer a key in sdk.tracer_provider.sampler_config. + # + # Environment variable: OTEL_TRACES_SAMPLER=parentbased_* + root: trace_id_ratio_based + # Set the sampler used when the parent is remote and is sampled. Must refer a key in sdk.tracer_provider.sampler_config. + remote_parent_sampled: always_on + # Set the sampler used when the parent is remote and is not sampled. Must refer a key in sdk.tracer_provider.sampler_config. + remote_parent_not_sampled: always_off + # Set the sampler used when the parent is local and is sampled. Must refer a key in sdk.tracer_provider.sampler_config. + local_parent_sampled: always_on + # Set the sampler used when the parent is local and is not sampled. Must refer a key in sdk.tracer_provider.sampler_config. + local_parent_not_sampled: always_off + # Configure the jaeger_remote sampler. + # + # Environment variable: OTEL_TRACES_SAMPLER=jaeger_remote + jaeger_remote: + # Set the endpoint. + # + # Environment variable: OTEL_TRACES_SAMPLER=jaeger_remote, OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:14250 + endpoint: http://localhost:14250 + # Set the polling interval. + # + # Environment variable: OTEL_TRACES_SAMPLER=jaeger_remote, OTEL_TRACES_SAMPLER_ARG=pollingINtervalMs=5000 + polling_interval: 5000 + # Set the initial sampling rate. + # + # Environment variable: OTEL_TRACES_SAMPLER=jaeger_remote, OTEL_TRACES_SAMPLER_ARG=initialSamplingRate=0.25 + initial_sampling_rate: 0.25 + # Set the sampler. Sampler must refer to a key in sdk.tracer_provider.sampler_config. + sampler: parent_based + # Configure the meter provider. + meter_provider: + # Metric exporters. Each exporter key refers to the type of the exporter. Values configure the exporter. Exporters must be associated with a metric reader. + exporters: + # Configure the otlp exporter. + otlp: + # Sets the protocol. + # + # Environment variable: OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL + protocol: http/protobuf + # Sets the endpoint. + # + # Environment variable: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT + endpoint: http://localhost:4318/v1/metrics + # Sets the certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE + certificate: /app/cert.pem + # Sets the mTLS private client key. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_KEY, OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY + client_key: /app/cert.pem + # Sets the mTLS client certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE, OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE + client_certificate: /app/cert.pem + # Sets the headers. + # + # Environment variable: OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_METRICS_HEADERS + headers: + api-key: 1234 + # Sets the compression. + # + # Environment variable: OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_METRICS_COMPRESSION + compression: gzip + # Sets the max time to wait for each export. + # + # Environment variable: OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_METRICS_TIMEOUT + timeout: 10000 + # Sets the temporality preference. + # + # Environment variable: OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE + temporality_preference: delta + # Sets the default histogram aggregation. + # + # Environment variable: OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION + default_histogram_aggregation: exponential_bucket_histogram + # List of metric readers. Each metric reader has a type and args used to configure it. + metric_readers: + # Add a periodic metric reader. + # + # Environment variable: OTEL_METRICS_EXPORT_*, OTEL_METRICS_EXPORTER + - type: periodic + args: + # Sets delay interval between the start of two consecutive export attempts. + # + # Environment variable: OTEL_METRIC_EXPORT_INTERVAL + interval: 5000 + # Sets the maximum allowed time to export data. + # + # Environment variable: OTEL_METRIC_EXPORT_TIMEOUT + timeout: 30000 + # Sets the exporter. Exporter must refer to a key in sdk.meter_provider.exporters. + # + # Environment variable: OTEL_METRICS_EXPORTER + exporter: otlp + # Add a prometheus metric reader. Some languages SDKs may implement this as a metric exporter. + # + # Environment variable: OTEL_METRICS_EXPORTER=prometheus + - type: prometheus + args: + # Set the host used to serve metrics in the prometheus format. + # + # Environment variable: OTEL_EXPORTER_PROMETHEUS_HOST + host: localhost + # Set the port used to serve metrics in the prometheus format. + # + # Environment variable: OTEL_EXPORTER_PROMETHEUS_PORT + port: 9464 + # List of views. Each view has a selector which determines the instrument(s) it applies to, and a view which configures resulting metric(s). + views: + # Add a view. The selection criteria and view aim to demonstrate the configuration surface area and are not representative of what a user would be expected to do. + - selector: + # Select instrument(s) by name. + instrument_name: my-instrument + # Select instrument(s) by type. + instrument_type: histogram + # Select instrument(s) by meter name. + meter_name: my-meter + # Select instrument(s) by meter version. + meter_version: 1.0.0 + # Select instrument(s) by meter schema URL. + meter_schema_url: https://opentelemetry.io/schemas/1.16.0 + view: + # Set the name of resulting metric(s). + name: new_instrument_name + # Set the description of resulting metric(s). + description: new_description + # Set the aggregation of resulting metric(s). Aggregation has a type an args used to configure it. + aggregation: + # Set the aggregation type. Options include: default, drop, sum, last_value, explicit_bucket_histogram, exponential_bucket_histogram. + type: explicit_bucket_histogram + # Configure the aggregation. + args: + # Set the bucket boundaries. Applicable when aggregation is explicit_bucket_histogram. + boundaries: [1.0, 2.0, 5.0] + # Set whether min and max are recorded. Applicable when aggregation is explicit_bucket_histogram or exponential_bucket_histogram. + record_min_max: true + # Sets the max number of buckets in each of the positive and negative ranges. Applicable when aggregation is exponential_bucket_histogram. + max_size: 160 + # Set the attribute keys to retain. + attribute_keys: + - key1 + - key2 + # Configure the logger provider. + logger_provider: + # Log record exporters. Each exporter key refers to the type of the exporter. Values configure the exporter. Exporters must be associated with a log record processor. + exporters: + # Configure the otlp exporter. + otlp: + # Sets the protocol. + # + # Environment variable: OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_LOGS_PROTOCOL + protocol: http/protobuf + # Sets the endpoint. + # + # Environment variable: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT + endpoint: http://localhost:4318/v1/logs + # Sets the certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE + certificate: /app/cert.pem + # Sets the mTLS private client key. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_KEY, OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY + client_key: /app/cert.pem + # Sets the mTLS client certificate. + # + # Environment variable: OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE, OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE + client_certificate: /app/cert.pem + # Sets the headers. + # + # Environment variable: OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_LOGS_HEADERS + headers: + api-key: 1234 + # Sets the compression. + # + # Environment variable: OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_LOGS_COMPRESSION + compression: gzip + # Sets the max time to wait for each export. + # + # Environment variable: OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_LOGS_TIMEOUT + timeout: 10000 + # List of log record processors. Each log record processor has a type and args used to configure it. + log_record_processors: + # Add a batch log record processor. + # + # Environment variable: OTEL_BLRP_*, OTEL_LOGS_EXPORTER + - type: batch + # Configure the batch log record processor. + args: + # Sets the delay interval between two consecutive exports. + # + # Environment variable: OTEL_BLRP_SCHEDULE_DELAY + schedule_delay: 5000 + # Sets the maximum allowed time to export data. + # + # Environment variable: OTEL_BLRP_EXPORT_TIMEOUT + export_timeout: 30000 + # Sets the maximum queue size. + # + # Environment variable: OTEL_BLRP_MAX_QUEUE_SIZE + max_queue_size: 2048 + # Sets the maximum batch size. + # + # Environment variable: OTEL_BLRP_MAX_EXPORT_BATCH_SIZE + max_export_batch_size: 512 + # Sets the exporter. Exporter must refer to a key in sdk.loger_provider.exporters. + # + # Environment variable: OTEL_LOGS_EXPORTER + exporter: otlp + # Configure the log record limits. See also sdk.attribute_limits. + log_record_limits: + # Set the max log record attribute value size. Overrides sdk.attribute_limits.attribute_value_length_limit. + # + # Environment variable: OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT + attribute_value_length_limit: 4096 + # Set the max log record attribute count. Overrides sdk.attribute_limits.attribute_count_limit. + # + # Environment variable: OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT + attribute_count_limit: 128 diff --git a/text/assets/0225-schema.json b/text/assets/0225-schema.json new file mode 100644 index 000000000..76af51b43 --- /dev/null +++ b/text/assets/0225-schema.json @@ -0,0 +1,656 @@ +{ + "$schema": "https://json-schema.org/draft-06/schema", + "$ref": "#/definitions/OpenTelemetryConfiguration", + "definitions": { + "OpenTelemetryConfiguration": { + "type": "object", + "additionalProperties": false, + "properties": { + "scheme_version": { + "type": "number" + }, + "sdk": { + "$ref": "#/definitions/SDK" + } + }, + "required": [ + "scheme_version", + "sdk" + ], + "title": "OpenTelemetryConfiguration" + }, + "SDK": { + "type": "object", + "additionalProperties": false, + "properties": { + "disabled": { + "type": "boolean" + }, + "resource": { + "$ref": "#/definitions/Resource" + }, + "propagators": { + "type": "array", + "items": { + "type": "string" + } + }, + "attribute_limits": { + "$ref": "#/definitions/Limits" + }, + "tracer_provider": { + "$ref": "#/definitions/TracerProvider" + }, + "meter_provider": { + "$ref": "#/definitions/MeterProvider" + }, + "logger_provider": { + "$ref": "#/definitions/LoggerProvider" + } + }, + "required": [ + "disabled" + ], + "title": "SDK" + }, + "Limits": { + "type": "object", + "additionalProperties": false, + "properties": { + "attribute_value_length_limit": { + "type": "integer" + }, + "attribute_count_limit": { + "type": "integer" + } + }, + "required": [ + "attribute_count_limit", + "attribute_value_length_limit" + ], + "title": "Limits" + }, + "LoggerProvider": { + "type": "object", + "additionalProperties": false, + "properties": { + "exporters": { + "$ref": "#/definitions/LoggerProviderExporters" + }, + "log_record_processors": { + "type": "array", + "items": { + "$ref": "#/definitions/Processor" + } + }, + "log_record_limits": { + "$ref": "#/definitions/Limits" + } + }, + "required": [ + "exporters", + "log_record_limits", + "log_record_processors" + ], + "title": "LoggerProvider" + }, + "LoggerProviderExporters": { + "type": "object", + "additionalProperties": false, + "properties": { + "otlp": { + "$ref": "#/definitions/Otlp" + } + }, + "required": [ + "otlp" + ], + "title": "LoggerProviderExporters" + }, + "Otlp": { + "type": "object", + "additionalProperties": false, + "properties": { + "protocol": { + "type": "string", + "pattern": "^(http|grpc)\\/(protobuf|json)" + }, + "endpoint": { + "type": "string", + "format": "uri", + "qt-uri-protocols": [ + "http" + ] + }, + "certificate": { + "type": "string" + }, + "client_key": { + "type": "string" + }, + "client_certificate": { + "type": "string" + }, + "headers": { + "$ref": "#/definitions/Headers" + }, + "compression": { + "type": "string" + }, + "timeout": { + "type": "integer" + }, + "temporality_preference": { + "type": "string" + }, + "default_histogram_aggregation": { + "type": "string" + } + }, + "required": [ + "endpoint", + "protocol" + ], + "title": "Otlp" + }, + "Headers": { + "type": "object", + "additionalProperties": true, + "title": "Headers" + }, + "Processor": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string" + }, + "args": { + "$ref": "#/definitions/LogRecordProcessorArgs" + } + }, + "required": [ + "args", + "type" + ], + "title": "Processor" + }, + "LogRecordProcessorArgs": { + "type": "object", + "additionalProperties": false, + "properties": { + "schedule_delay": { + "type": "integer" + }, + "export_timeout": { + "type": "integer" + }, + "max_queue_size": { + "type": "integer" + }, + "max_export_batch_size": { + "type": "integer" + }, + "exporter": { + "type": "string" + } + }, + "required": [ + "exporter" + ], + "title": "LogRecordProcessorArgs" + }, + "MeterProvider": { + "type": "object", + "additionalProperties": false, + "properties": { + "exporters": { + "$ref": "#/definitions/LoggerProviderExporters" + }, + "metric_readers": { + "type": "array", + "items": { + "$ref": "#/definitions/MetricReader" + } + }, + "views": { + "type": "array", + "items": { + "$ref": "#/definitions/ViewElement" + } + } + }, + "required": [ + "exporters", + "metric_readers", + "views" + ], + "title": "MeterProvider" + }, + "MetricReader": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string" + }, + "args": { + "$ref": "#/definitions/MetricReaderArgs" + } + }, + "required": [ + "args", + "type" + ], + "title": "MetricReader" + }, + "MetricReaderArgs": { + "type": "object", + "additionalProperties": false, + "properties": { + "interval": { + "type": "integer" + }, + "timeout": { + "type": "integer" + }, + "exporter": { + "type": "string" + }, + "host": { + "type": "string" + }, + "port": { + "type": "integer" + } + }, + "required": [], + "title": "MetricReaderArgs" + }, + "ViewElement": { + "type": "object", + "additionalProperties": false, + "properties": { + "selector": { + "$ref": "#/definitions/Selector" + }, + "view": { + "$ref": "#/definitions/ViewView" + } + }, + "required": [ + "selector", + "view" + ], + "title": "ViewElement" + }, + "Selector": { + "type": "object", + "additionalProperties": false, + "properties": { + "instrument_name": { + "type": "string" + }, + "instrument_type": { + "type": "string" + }, + "meter_name": { + "type": "string" + }, + "meter_version": { + "type": "string" + }, + "meter_schema_url": { + "type": "string", + "format": "uri", + "qt-uri-protocols": [ + "https" + ], + "qt-uri-extensions": [ + ".0" + ] + } + }, + "required": [ + "instrument_name", + "instrument_type", + "meter_name", + "meter_schema_url", + "meter_version" + ], + "title": "Selector" + }, + "ViewView": { + "type": "object", + "additionalProperties": false, + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "aggregation": { + "$ref": "#/definitions/Aggregation" + }, + "attribute_keys": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "aggregation", + "attribute_keys", + "description", + "name" + ], + "title": "ViewView" + }, + "Aggregation": { + "type": "object", + "additionalProperties": false, + "properties": { + "type": { + "type": "string" + }, + "args": { + "$ref": "#/definitions/AggregationArgs" + } + }, + "required": [ + "args", + "type" + ], + "title": "Aggregation" + }, + "AggregationArgs": { + "type": "object", + "additionalProperties": false, + "properties": { + "boundaries": { + "type": "array", + "items": { + "type": "number" + } + }, + "record_min_max": { + "type": "boolean" + }, + "max_size": { + "type": "integer" + } + }, + "required": [ + "boundaries", + "max_size", + "record_min_max" + ], + "title": "AggregationArgs" + }, + "Resource": { + "type": "object", + "additionalProperties": false, + "properties": { + "attributes": { + "$ref": "#/definitions/Attributes" + } + }, + "required": [ + "attributes" + ], + "title": "Resource" + }, + "Attributes": { + "type": "object", + "additionalProperties": true, + "properties": { + "service.name": { + "type": "string" + } + }, + "required": [ + "service.name" + ], + "title": "Attributes" + }, + "TracerProvider": { + "type": "object", + "additionalProperties": false, + "properties": { + "exporters": { + "$ref": "#/definitions/TracerProviderExporters" + }, + "span_processors": { + "type": "array", + "items": { + "$ref": "#/definitions/Processor" + } + }, + "span_limits": { + "$ref": "#/definitions/SpanLimits" + }, + "sampler_config": { + "$ref": "#/definitions/SamplerConfig" + }, + "sampler": { + "type": "string" + } + }, + "required": [ + "exporters", + "span_processors" + ], + "title": "TracerProvider" + }, + "TracerProviderExporters": { + "type": "object", + "additionalProperties": true, + "patternProperties": { + "^otlp.*": { + "$ref": "#/definitions/Otlp" + }, + "^zipkin.*": { + "$ref": "#/definitions/Zipkin" + }, + "^jaeger.*": { + "$ref": "#/definitions/Jaeger" + } + }, + "title": "TracerProviderExporters" + }, + "Jaeger": { + "type": "object", + "additionalProperties": false, + "properties": { + "protocol": { + "type": "string", + "pattern": "^http/thrift.binary$" + }, + "endpoint": { + "type": "string", + "format": "uri", + "qt-uri-protocols": [ + "http" + ] + }, + "timeout": { + "type": "integer" + }, + "user": { + "type": "string" + }, + "password": { + "type": "string" + }, + "agent_host": { + "type": "string" + }, + "agent_port": { + "type": "integer" + } + }, + "required": [ + "agent_host", + "agent_port", + "endpoint", + "password", + "protocol", + "timeout", + "user" + ], + "title": "Jaeger" + }, + "Zipkin": { + "type": "object", + "additionalProperties": false, + "properties": { + "endpoint": { + "type": "string", + "format": "uri", + "qt-uri-protocols": [ + "http" + ] + }, + "timeout": { + "type": "integer" + } + }, + "required": [ + "endpoint", + "timeout" + ], + "title": "Zipkin" + }, + "SamplerConfig": { + "type": "object", + "additionalProperties": false, + "properties": { + "always_on": { + "type": "null" + }, + "always_off": { + "type": "null" + }, + "trace_id_ratio_based": { + "$ref": "#/definitions/TraceIDRatioBased" + }, + "parent_based": { + "$ref": "#/definitions/ParentBased" + }, + "jaeger_remote": { + "$ref": "#/definitions/JaegerRemote" + } + }, + "required": [ + "always_off", + "always_on", + "jaeger_remote", + "parent_based", + "trace_id_ratio_based" + ], + "title": "SamplerConfig" + }, + "JaegerRemote": { + "type": "object", + "additionalProperties": false, + "properties": { + "endpoint": { + "type": "string", + "format": "uri", + "qt-uri-protocols": [ + "http" + ] + }, + "polling_interval": { + "type": "integer" + }, + "initial_sampling_rate": { + "type": "number" + } + }, + "required": [ + "endpoint", + "initial_sampling_rate", + "polling_interval" + ], + "title": "JaegerRemote" + }, + "ParentBased": { + "type": "object", + "additionalProperties": false, + "properties": { + "root": { + "type": "string" + }, + "remote_parent_sampled": { + "type": "string" + }, + "remote_parent_not_sampled": { + "type": "string" + }, + "local_parent_sampled": { + "type": "string" + }, + "local_parent_not_sampled": { + "type": "string" + } + }, + "required": [ + "local_parent_not_sampled", + "local_parent_sampled", + "remote_parent_not_sampled", + "remote_parent_sampled", + "root" + ], + "title": "ParentBased" + }, + "TraceIDRatioBased": { + "type": "object", + "additionalProperties": false, + "properties": { + "ratio": { + "type": "number" + } + }, + "required": [ + "ratio" + ], + "title": "TraceIDRatioBased" + }, + "SpanLimits": { + "type": "object", + "additionalProperties": false, + "properties": { + "attribute_value_length_limit": { + "type": "integer" + }, + "attribute_count_limit": { + "type": "integer" + }, + "event_count_limit": { + "type": "integer" + }, + "link_count_limit": { + "type": "integer" + }, + "event_attribute_count_limit": { + "type": "integer" + }, + "link_attribute_count_limit": { + "type": "integer" + } + }, + "required": [ + "attribute_count_limit", + "attribute_value_length_limit", + "event_attribute_count_limit", + "event_count_limit", + "link_attribute_count_limit", + "link_count_limit" + ], + "title": "SpanLimits" + } + } +} \ No newline at end of file