Telemetry Manager

Overview

To implement Kyma's strategy of moving from in-cluster observability backends to a Telemetry component that integrates with external backends, the telemetry operator provides APIs for configurable logging, tracing, and monitoring.

The telemetry operator has been bootstrapped with Kubebuilder 3.6.0. Additional APIs can also be added by Kubebuilder.

Configurable Logging

The logging controllers generate a Fluent Bit configuration from one or more LogPipeline and LogParser custom resources. The controllers ensure that all Fluent Bit Pods run the current configuration by restarting Pods after the configuration has changed. See all CRD attributes and some examples.

For now, creating Fluent Bit Pods is out of scope of the operator. An existing Fluent Bit DaemonSet is expected.

The generated ConfigMap (by default, telemetry-fluent-bit-sections in the kyma-system namespace) must be mounted to the Fluent Bit Pods and consumed by an @INCLUDE statement in an existing configuration file. Fluent Bit parsers, file references, and environment variables are available in an additional ConfigMap or Secret.

See the flags that configure all ConfigMaps, Secret and DaemonSet names in main.go.

Further design decisions and test results are documented in Dynamic Logging Backend Configuration.

Configurable Tracing

Configurable tracing is still in development and not active with the default Kyma settings.

The trace controller creates an OpenTelemetry Collector deployment and related Kubernetes objects from a TracePipeline custom resource. The collector is configured to receive traces using the OTLP and OpenCensus protocols, and forwards the received traces to a configurable OTLP backend.

See Dynamic Trace Backend Configuration for further information.

Configurable Monitoring

Configurable monitoring is not implemented yet. Future plans are documented in Dynamic Monitoring Backend Configuration.

Development

Prerequisites

• Install kubebuilder 3.6.0, which is the base framework for this controller.
• Install kustomize which lets you customize raw, template-free YAML files during local development.
• Install Golang 1.19 or newer (for local execution).
• Install Docker.
• Install OpenSSL to generate a webhook certificate for local execution.

Available Commands

For development, you can use the following commands:

• Run all tests and validation

make

• Regenerate YAML manifests (CRDs and ClusterRole)

make manifests

• Install CRDs to cluster in current kubeconfig context

make install

• Uninstall CRDs to cluster in current kubeconfig context

make uninstall

• Run the operator locally (uses current kubeconfig context)

kubectl -n kyma-system scale deployment telemetry-operator --replicas=0 # Scale down in-cluster telemetry-operator
make run

• Build container image and deploy to cluster in current kubeconfig context. Deploy telemetry chart first, as described before. Then run the following commands to deploy your own operator image.

make docker-build IMG=<my container repo>
make docker-push IMG=<my container repo>
kubectl -n kyma-system set image deployment telemetry-operator manager=<my container repo>

Troubleshooting

Enable pausing reconciliations

You must pause reconciliations to be able to debug the pipelines and, for example, try out a different pipeline configuration or a different OTel configuration. To pause reconciliations, create a telemetry-override-config in the kyma-system Namespace. Here is an example of such a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: telemetry-override-config
  namespace: kyma-system
data:
  override-config: |
    global:
      logLevel: debug
    tracing:
      paused: true
    logging:
      paused: true

The global, tracing, and logging fields are optional.

Debugging steps

1. Create an overriding telemetry-override-config ConfigMap.
2. Perform debugging operations.
3. Remove the created ConfigMap.
4. To reset the debug actions, perform a restart of the telemetry operator.

   kubectl rollout restart deployment -n kyma-system telemetry-operator

Caveats If you change the pipeline CR when the reconciliation is paused, these changes will not be applied immediately but in a periodic reconciliation cycle of one hour. To reconcile earlier, restart the Telemetry operator.

Profiling

The Telemetry operator has pprof-based profiling activated and exposed on port 6060. Use port-forwarding to access the pprof endpoint. You can find additional information in the Go pprof package documentation.