OpenTelemetry reported "unknown variable" before first Ingress

[mtslzr]

What happened:

While doing some Disaster Recovery testing, we re-deployed ingress-nginx with the following http-snippet to log trace IDs (pulled from opentracing-contrib/nginx-opentracing#33):

http-snippet: |
        map $opentelemetry_context_traceparent $ot_trace_id {
          default "00000000000000000000000000000000";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$trace_id";
        }
        map $opentelemetry_context_traceparent $ot_span_id {
          default "0000000000000000";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$parent_id";
        }
        map $opentelemetry_context_traceparent $ot_trace_flags {
          default "00";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$trace_flags";
        }

When it spins up, until the first Ingress is created, the opentelemetry_ variables are all unknown, resulting in the following error:

Error: exit status 1
2024/02/29 15:55:17 [emerg] 26#26: unknown "opentelemetry_context_traceparent" variable
nginx: [emerg] unknown "opentelemetry_context_traceparent" variable
nginx: configuration file /tmp/nginx/nginx-cfg966897373 test failed

What you expected to happen:

I would prefer that setting the following makes the opentelemetry_ variable available from the start, instead of waiting for the first Ingress:

opentelemetry:
  enabled: true

Alternatively, I've tried figuring out a workaround to just ignore the variable if unavailable using map and could not find anything workable, and know that's likely outside the scope of this issue.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
  Release:       v1.9.6
  Build:         6a73aa3b05040a97ef8213675a16142a9c95952a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

(This is from testing locally with minikube)

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Environment:

• Cloud provider or hardware configuration: Azure / testing locally with minikube
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: Rendered locally/via CI with helm template and then applied locally or pushed to a control repo and consumed by Argo
  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
• Basic cluster related info:
  • kubectl version
  • kubectl get nodes -o wide

❯ kubectl get nodes -o wide
NAME                              STATUS   ROLES   AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-default-vmss0004rv   Ready    agent   2d13h   v1.28.5   10.1.10.179   <none>        Ubuntu 22.04.3 LTS   5.15.0-1054-azure   containerd://1.7.7-1
aks-default-vmss0004tm   Ready    agent   2d13h   v1.28.5   10.1.1.45     <none>        Ubuntu 22.04.3 LTS   5.15.0-1054-azure   containerd://1.7.7-1

• How was the ingress-nginx-controller installed:

Rendered using helm template either locally for testing or via Gitlab CI, and then pushed to a controller repo and consumed by Argo (or applied locally to minikube for testing).

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.9.6
              helm.sh/chart=ingress-nginx-4.9.1
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

NAMESPACE     NAME                                               READY   STATUS    RESTARTS        AGE   IP             NODE       NOMINATED NODE   READINESS GATES
ingress       pod/nginx-ingress-ingress-nginx-controller-4sdxg   0/1     Running   8 (5m38s ago)   18m   10.244.0.17    minikube   <none>           <none>
kube-system   pod/coredns-5dd5756b68-pkrjg                       1/1     Running   0               13d   10.244.0.2     minikube   <none>           <none>
kube-system   pod/etcd-minikube                                  1/1     Running   0               13d   192.168.49.2   minikube   <none>           <none>
kube-system   pod/kube-apiserver-minikube                        1/1     Running   0               13d   192.168.49.2   minikube   <none>           <none>
kube-system   pod/kube-controller-manager-minikube               1/1     Running   0               13d   192.168.49.2   minikube   <none>           <none>
kube-system   pod/kube-proxy-6rb6q                               1/1     Running   0               13d   192.168.49.2   minikube   <none>           <none>
kube-system   pod/kube-scheduler-minikube                        1/1     Running   0               13d   192.168.49.2   minikube   <none>           <none>
kube-system   pod/storage-provisioner                            1/1     Running   1 (13d ago)     13d   192.168.49.2   minikube   <none>           <none>

NAMESPACE     NAME                                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default       service/kubernetes                                         ClusterIP      10.96.0.1        <none>        443/TCP                      13d   <none>
ingress       service/nginx-ingress-ingress-nginx-controller             LoadBalancer   10.108.108.1     <pending>     80:30355/TCP,443:30322/TCP   8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
ingress       service/nginx-ingress-ingress-nginx-controller-admission   ClusterIP      10.100.100.183   <none>        443/TCP                      8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
ingress       service/nginx-ingress-ingress-nginx-controller-metrics     ClusterIP      10.103.113.179   <none>        10254/TCP                    8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
kube-system   service/kube-dns                                           ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       13d   k8s-app=kube-dns

NAMESPACE     NAME                                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
ingress       daemonset.apps/nginx-ingress-ingress-nginx-controller   1         1         0       1            0           kubernetes.io/os=linux   8d    controller   registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
kube-system   daemonset.apps/kube-proxy                               1         1         1       1            1           kubernetes.io/os=linux   13d   kube-proxy   registry.k8s.io/kube-proxy:v1.28.3                                                                                        k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                    SELECTOR
kube-system   deployment.apps/coredns   1/1     1            1           13d   coredns      registry.k8s.io/coredns/coredns:v1.10.1   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                    SELECTOR
kube-system   replicaset.apps/coredns-5dd5756b68   1         1         1       13d   coredns      registry.k8s.io/coredns/coredns:v1.10.1   k8s-app=kube-dns,pod-template-hash=5dd5756b68

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Namespace:            ingress
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      nginx-ingress-ingress-nginx
Node:                 minikube/192.168.49.2
Start Time:           Thu, 29 Feb 2024 15:55:09 +0000
Labels:               app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=nginx-ingress
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.9.6
                    controller-revision-hash=5454569bdb
                    helm.sh/chart=ingress-nginx-4.9.1
                    pod-template-generation=4
Annotations:          cluster-autoscaler.kubernetes.io/enable-ds-eviction: true
                    prometheus.io/port: 10254
                    prometheus.io/scrape: true
Status:               Running
IP:                   10.244.0.17
IPs:
IP:           10.244.0.17
Controlled By:  DaemonSet/nginx-ingress-ingress-nginx-controller
Init Containers:
opentelemetry:
  Container ID:    docker://f88203da0588900839edb6a10b2937922809f5159d990e5917177b3c315f81a3
  Image:           registry.k8s.io/ingress-nginx/opentelemetry:v20230721-3e2062ee5@sha256:13bee3f5223883d3ca62fee7309ad02d22ec00ff0d7033e3e9aca7a9f60fd472
  Image ID:        docker-pullable://registry.k8s.io/ingress-nginx/opentelemetry@sha256:13bee3f5223883d3ca62fee7309ad02d22ec00ff0d7033e3e9aca7a9f60fd472
  Port:            <none>
  Host Port:       <none>
  SeccompProfile:  RuntimeDefault
  Command:
    /init_module
  State:          Terminated
    Reason:       Completed
    Exit Code:    0
    Started:      Thu, 29 Feb 2024 15:55:09 +0000
    Finished:     Thu, 29 Feb 2024 15:55:09 +0000
  Ready:          True
  Restart Count:  0
  Environment:    <none>
  Mounts:
    /modules_mount from modules (rw)
Containers:
controller:
  Container ID:    docker://80963e4ef90bfad245ac1b8bd4e8c037f3e5f871fa10b6bfcc0462b362d4e24e
  Image:           registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c
  Image ID:        docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c
  Ports:           80/TCP, 443/TCP, 10254/TCP, 8443/TCP
  Host Ports:      0/TCP, 0/TCP, 0/TCP, 0/TCP
  SeccompProfile:  RuntimeDefault
  Args:
    /nginx-ingress-controller
    --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
    --election-id=nginx-ingress-ingress-nginx-leader
    --controller-class=k8s.io/ingress-nginx
    --ingress-class=nginx
    --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
    --metrics-per-host=false
  State:          Running
    Started:      Thu, 29 Feb 2024 16:13:33 +0000
  Last State:     Terminated
    Reason:       Completed
    Exit Code:    0
    Started:      Thu, 29 Feb 2024 16:07:09 +0000
    Finished:     Thu, 29 Feb 2024 16:08:22 +0000
  Ready:          False
  Restart Count:  8
  Requests:
    cpu:      100m
    memory:   190Mi
  Liveness:   http-get http://:10254/health delay=10s timeout=1s period=10s #success=1 #failure=5
  Readiness:  http-get http://:10254/health delay=10s timeout=1s period=10s #success=1 #failure=3
  Environment:
    POD_NAME:       nginx-ingress-ingress-nginx-controller-4sdxg (v1:metadata.name)
    POD_NAMESPACE:  ingress (v1:metadata.namespace)
    LD_PRELOAD:     /usr/local/lib/libmimalloc.so
  Mounts:
    /modules_mount from modules (rw)
Conditions:
Type              Status
Initialized       True
Ready             False
ContainersReady   False
PodScheduled      True
Volumes:
modules:
  Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
  Medium:
  SizeLimit:  <unset>
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                           node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                           node.kubernetes.io/not-ready:NoExecute op=Exists
                           node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                           node.kubernetes.io/unreachable:NoExecute op=Exists
                           node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type     Reason     Age   From                      Message
----     ------     ----  ----                      -------
Normal   Scheduled  19m   default-scheduler         Successfully assigned ingress/nginx-ingress-ingress-nginx-controller-4sdxg to minikube
Normal   Pulled     19m   kubelet                   Container image "registry.k8s.io/ingress-nginx/opentelemetry:v20230721-3e2062ee5@sha256:13bee3f5223883d3ca62fee7309ad02d22ec00ff0d7033e3e9aca7a9f60fd472" already present on machine
Normal   Created    19m   kubelet                   Created container opentelemetry
Normal   Started    19m   kubelet                   Started container opentelemetry
Warning  RELOAD     19m   nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2024/02/29 15:55:11 [emerg] 23#23: unknown "opentelemetry_context_traceparent" variable
nginx: [emerg] unknown "opentelemetry_context_traceparent" variable
nginx: configuration file /tmp/nginx/nginx-cfg1267461679 test failed

-------------------------------------------------------------------------------
Warning  RELOAD  19m  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2024/02/29 15:55:14 [emerg] 25#25: unknown "opentelemetry_context_traceparent" variable
nginx: [emerg] unknown "opentelemetry_context_traceparent" variable
nginx: configuration file /tmp/nginx/nginx-cfg923001734 test failed

-------------------------------------------------------------------------------
Warning  RELOAD  19m  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2024/02/29 15:55:17 [emerg] 26#26: unknown "opentelemetry_context_traceparent" variable
nginx: [emerg] unknown "opentelemetry_context_traceparent" variable
nginx: configuration file /tmp/nginx/nginx-cfg966897373 test failed

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     nginx-ingress-ingress-nginx-controller
Namespace:                ingress
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=nginx-ingress
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.9.6
                          helm.sh/chart=ingress-nginx-4.9.1
Annotations:              service.beta.kubernetes.io/azure-load-balancer-resource-group: kubernetes-development
                          service.beta.kubernetes.io/azure-pip-name: nginx-ingress-pip
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.108.108.1
IPs:                      10.108.108.1
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30355/TCP
Endpoints:
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30322/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32131
Events:                   <none>

• Current state of ingress object, if applicable:
  • kubectl -n <appnamespace> get all,ing -o wide

NAME                                               READY   STATUS    RESTARTS      AGE   IP            NODE       NOMINATED NODE   READINESS GATES
pod/nginx-ingress-ingress-nginx-controller-4sdxg   0/1     Running   9 (80s ago)   21m   10.244.0.17   minikube   <none>           <none>

NAME                                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
service/nginx-ingress-ingress-nginx-controller             LoadBalancer   10.108.108.1     <pending>     80:30355/TCP,443:30322/TCP   8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-ingress-nginx-controller-admission   ClusterIP      10.100.100.183   <none>        443/TCP                      8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-ingress-nginx-controller-metrics     ClusterIP      10.103.113.179   <none>        10254/TCP                    8d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

NAME                                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
daemonset.apps/nginx-ingress-ingress-nginx-controller   1         1         0       1            0           kubernetes.io/os=linux   8d    controller   registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

• kubectl -n <appnamespace> describe ing <ingressname>

• If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Install minikube locally. Set up the following Helm chart:

apiVersion: v2
name: my-nginx-ingress
description: Nginx Ingress
type: application
dependencies:
  - name: ingress-nginx
    version: 4.9.1
    repository: https://kubernetes.github.io/ingress-nginx

And set the following in values.yaml:

ingress-nginx:
  controller:
    ...
    opentelemetry:
      enabled: true

      # from: https://github.com/opentracing-contrib/nginx-opentracing/issues/33
      http-snippet: |
        map $opentelemetry_context_traceparent $ot_trace_id {
          default "00000000000000000000000000000000";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$trace_id";
        }
        map $opentelemetry_context_traceparent $ot_span_id {
          default "0000000000000000";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$parent_id";
        }
        map $opentelemetry_context_traceparent $ot_trace_flags {
          default "00";
          ~^00-(?<trace_id>[^-]+)-(?<parent_id>[^-]+)-(?<trace_flags>[0-9]+)$ "$trace_flags";
        }

Render it went helm template and then kube apply -f. When the pod comes up, kubectl logs to see the above errors.

Anything else we need to know:

Not sure if the request is nonsensical, so open to a work-around to just... ignore the issue until an Ingress is created, but it's been a bit of an annoyance during testing.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/assign @esigo

[esigo]

we can't set the context without a request. It's meaningless to have the variable set.you'd need to find a workaround. Maybe sth similar to #9811 (comment).

[longwuyuan]

/remove-kind bug
/kind support

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.