Annotations Filter does not Filter Annotations

[ryanohnemus]

Component(s)

collector

What happened?

Description

When using an annotations filter with the opentelemetry-operator the annotations are not actually being filtered out in the Deployment that is created.

Steps to Reproduce

I setup a kind cluster using the kind-1.29.yaml included in the repo:

# setup kind
kind create cluster --config kind-1.29.yaml

# add required cert-manager install
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

# Download the latest operator install
curl -Lo otel-op.yaml https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

I then edited the otel-op.yaml to add the --annotations-filter="config.*.gke.io.*" arg below for the operator's Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: opentelemetry-operator
    control-plane: controller-manager
  name: opentelemetry-operator-controller-manager
  namespace: opentelemetry-operator-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-operator
      control-plane: controller-manager
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetry-operator
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --zap-log-level=info
        - --zap-time-encoding=rfc3339nano
        - --enable-nginx-instrumentation=true
        - --annotations-filter="config.*.gke.io.*"
        env:
        - name: SERVICE_ACCOUNT_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceAccountName
        image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.101.0

I then applied operator to the cluster and confirmed the operator pod args contained the annotations-filter above:
kubectl apply -f otel-op.yaml
Finally I added an OpenTelemetryCollector resource that has an annotation configmanagement.gke.io/token I was expecting filtered out:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
  annotations:
    configmanagement.gke.io/token: "asdfasdf"
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [debug]

The logs below show opentelemetry-operator the annotation-filter was loaded

Expected Result

The configmanagement.gke.io/token annotation does not make it to the collector deployment created by otel-operator.

Actual Result

The configmanagement.gke.io/token: asdfasdf is still added to the pod template in the deployment of the collector and is still added to the pod.

Kubernetes Version

1.29.0

Operator version

0.101.0

Collector version

0.101.0

Environment information

Environment

kind cluster 1.29 (already included in repo)

Log output

# OpenTelemetry Operator shows the annotation filter was loaded

{"level":"INFO","timestamp":"2024-06-03T16:18:48.225161474Z","message":"Starting the OpenTelemetry Operator","opentelemetry-operator":"0.101.0","opentelemetry-collector":"ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.101.0","opentelemetry-targetallocator":"ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.101.0","operator-opamp-bridge":"ghcr.io/open-telemetry/opentelemetry-operator/operator-opamp-bridge:0.101.0","auto-instrumentation-java":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.32.1","auto-instrumentation-nodejs":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.51.0","auto-instrumentation-python":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0","auto-instrumentation-dotnet":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.2.0","auto-instrumentation-go":"ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.12.0-alpha","auto-instrumentation-apache-httpd":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","auto-instrumentation-nginx":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","feature-gates":"-operator.golang.flags,operator.observability.prometheus","build-date":"2024-05-29T12:23:02Z","go-version":"go1.21.10","go-arch":"amd64","go-os":"linux","labels-filter":[],"annotations-filter":["\"config.*.gke.io.*\""],"enable-multi-instrumentation":false,"enable-apache-httpd-instrumentation":true,"enable-dotnet-instrumentation":true,"enable-go-instrumentation":false,"enable-python-instrumentation":true,"enable-nginx-instrumentation":true,"enable-nodejs-instrumentation":true,"enable-java-instrumentation":true,"zap-message-key":"message","zap-level-key":"level","zap-time-key":"timestamp","zap-level-format":"uppercase"}

Additional context

No response

[jaronoff97]

Taking a look at this now

[jaronoff97]

Okay, i was able to repro this,

⫸ k describe pod simplest-collector-758f8f96c4-hzm6c
Name:             simplest-collector-758f8f96c4-hzm6c
Namespace:        default
Priority:         0
Service Account:  simplest-collector
Node:             kind-control-plane/172.20.0.2
Start Time:       Mon, 03 Jun 2024 14:56:32 -0400
Labels:           app.kubernetes.io/component=opentelemetry-collector
                  app.kubernetes.io/instance=default.simplest
                  app.kubernetes.io/managed-by=opentelemetry-operator
                  app.kubernetes.io/name=simplest-collector
                  app.kubernetes.io/part-of=opentelemetry
                  app.kubernetes.io/version=latest
                  pod-template-hash=758f8f96c4
Annotations:      configmanagement.gke.io/token: asdfasdf

But, I thought this is actually because it's adding in the supplied double quotes: ,"annotations-filter":["\"config.*.gke.io.*\""]

Removing that and reapplying it:

- --annotations-filter=config.*.gke.io.*

I reapplied it and was wrong... onto some more debugging I go.

[jaronoff97]

Okay, i found the problem. It's because we escape the regex string you provide right here. This is why the unit test doesn't catch it, we don't apply it with the option pattern. Fixing this now.

[jaronoff97]

@ryanohnemus thanks for the report and the repro! I have a fix up (see above PR) that I will make sure to get merged prior to the next release.

[ryanohnemus]

Awesome!! This is extremely appreciated, thank you @jaronoff97!!

[jaronoff97]

For extra confirmation, i installed a local copy of the operator with this change, followed the repro steps and it's gone now 🎉

⫸ k describe pod simplest-collector-6bf5879bd-9jqx8
Name:             simplest-collector-6bf5879bd-9jqx8
Namespace:        default
Priority:         0
Service Account:  simplest-collector
Node:             kind-control-plane/172.20.0.2
Start Time:       Mon, 03 Jun 2024 15:34:40 -0400
Labels:           app.kubernetes.io/component=opentelemetry-collector
                  app.kubernetes.io/instance=default.simplest
                  app.kubernetes.io/managed-by=opentelemetry-operator
                  app.kubernetes.io/name=simplest-collector
                  app.kubernetes.io/part-of=opentelemetry
                  app.kubernetes.io/version=latest
                  pod-template-hash=6bf5879bd
Annotations:      opentelemetry-operator-config/sha256: cae1dd178ddb06dcda2011f11e7e4e4408b5b42c5195e63edca71bed1e99d1c4
                  prometheus.io/path: /metrics
                  prometheus.io/port: 8888
                  prometheus.io/scrape: true