[BUG] K8S env variable not loaded

[schmittse]

What happened?

I've deployed a bunkerweb Ingress Controller using the documentation.
It leaded to a working DeamonSet : (only usefull part of yaml copied here)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bunkerweb
  namespace: bunkerweb
spec:
  [...]
    spec:
      containers:
      - env:
        - name: KUBERNETES_MODE
          value: "yes"
        - name: MULTISITE
          value: "yes"
        - name: USE_REDIS
          value: "yes"
        - name: REDIS_HOST
          value: svc-bunkerweb-redis
        image: bunkerity/bunkerweb:1.5.10
        name: bunkerweb

But is the outpu log, we can see those lines :

2024/10/28 08:33:23 [error] 2318276#2318276: *740820 [BADBEHAVIOR] (increase) redis_increase failed, falling back to local : error while getting redis client : failed to parse host name "": no host, context: ngx.timer, client: 1.2.3.4, server: 0.0.0.0:8080

And if we log inside the pod and look at the variables file, we can see that the REDIS host is empty :

$ cat /etc/nginx/variables.env | grep "REDIS"
USE_REDIS=yes
REDIS_HOST=
REDIS_PORT=6379
REDIS_DATABASE=0
REDIS_SSL=no
REDIS_SSL_VERIFY=no
REDIS_TIMEOUT=1000
REDIS_USERNAME=
REDIS_PASSWORD=
REDIS_SENTINEL_HOSTS=
REDIS_SENTINEL_USERNAME=
REDIS_SENTINEL_PASSWORD=
REDIS_SENTINEL_MASTER=
REDIS_KEEPALIVE_IDLE=30000
REDIS_KEEPALIVE_POOL=10

I made this bug report using REDIS_HOST as exemple but this is true for a lot of env variables.

How to reproduce?

Install a fresh bunkerweb in a Kubernetes environnement.

Configuration file(s) (yaml or .env)

# Redis deployement :
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb-redis
  namespace: bunkerweb
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: bunkerweb-redis
  template:
    metadata:
      labels:
        app: bunkerweb-redis
    spec:
      containers:
        - name: bunkerweb-redis
          image: redis:7-alpine
          imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: svc-bunkerweb-redis
  namespace: bunkerweb
spec:
  type: ClusterIP
  selector:
    app: bunkerweb-redis
  ports:
    - name: redis
      protocol: TCP
      port: 6379
      targetPort: 6379
# Bunkerweb DaemonSet
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bunkerweb
  namespace: bunkerweb
spec:
  selector:
    matchLabels:
      app: bunkerweb
  template:
    metadata:
      labels:
        app: bunkerweb
      # mandatory annotation
      annotations:
        bunkerweb.io/INSTANCE: "yes"
    spec:
      serviceAccountName: sa-bunkerweb
      containers:
        # using bunkerweb as name is mandatory
        - name: bunkerweb
          image: bunkerity/bunkerweb:1.5.10
          imagePullPolicy: Always
          securityContext:
            runAsUser: 101
            runAsGroup: 101
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          ports:
            - containerPort: 8080
              name: htpp
            - containerPort: 8443
              name: https
          env:
            - name: KUBERNETES_MODE
              value: "yes"
            - name: DNS_RESOLVERS
              value: "coredns.kube-system.svc"
            - name: USE_API
              value: "yes"
            - name: API_WHITELIST_IP
              value: "127.0.0.0/8 10.0.0.0/8"
            - name: SERVER_NAME
              value: ""
            - name: MULTISITE
              value: "yes"
            - name: USE_REDIS
              value: "yes"
            - name: REDIS_HOST
              value: "svc-bunkerweb-redis"
            - name: USE_PROXY_PROTOCOL
              value: "yes"
            - name: USE_REAL_IP
              value: "yes"
            - name: REAL_IP_FROM
              value: "1.2.3.4/32"
          livenessProbe:
            exec:
              command:
                - /usr/share/bunkerweb/helpers/healthcheck.sh
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 3
          readinessProbe:
            exec:
              command:
                - /usr/share/bunkerweb/helpers/healthcheck.sh
            initialDelaySeconds: 30
            periodSeconds: 1
            timeoutSeconds: 1
            failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
  name: svc-bunkerweb
  namespace: bunkerweb
spec:
  clusterIP: None
  selector:
    app: bunkerweb

Relevant log output

hello-world.example.com 1.2.3.4 - - [28/Oct/2024:08:41:50 +0000] "GET /.well-known/acme-challenge/pK8hcNAZCqPFnq9qFNMx3vGDxhhbBUkS5CjZ84lcV_A HTTP/1.1" 404 89923 "-" "cert-manager-challenges/v1.13.6 (linux/amd64) cert-manager/d9235ef134b120ef5e7c7fc58d342e66599069b5"
2024/10/28 08:41:50 [error] 2318276#2318276: *742389 [BADBEHAVIOR] (increase) redis_increase failed, falling back to local : error while getting redis client : failed to parse host name "": no host, context: ngx.timer, client: 1.2.3.4, server: 0.0.0.0:8080
2024/10/28 08:41:50 [notice] 2318276#2318276: *742389 [BADBEHAVIOR] increased counter for IP 1.2.3.4 (2/10), context: ngx.timer, client: 1.2.3.4, server: 0.0.0.0:8080

BunkerWeb version

1.5.10

What integration are you using?

Kubernetes

Linux distribution (if applicable)

Ubuntu Server 20.04

Removed private data

• I have removed all private data from the configuration file and the logs

Code of Conduct

• I agree to follow this project's Code of Conduct

[TheophileDiot]

Hi @schmittse, did you try again with the 1.5.11 version?

[civts]

Hi @TheophileDiot, I can confirm that the issue is still present in the latest version, 1.5.12.

[TheophileDiot]

Hi @schmittse and @civts. Have you tried the latest release candidate ? Remember to checkout the documentation before hand: https://docs.bunkerweb.io/1.6.0-rc2/integrations/

[civts]

Yes, the issue is present also in version 1.6.0-rc2 (nice UI update by the way!)

To reproduce, deploy a bunkerweb instance in an empty cluster (I used k8s version 1.31.0) -example taken from the documentation-

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bunkerweb
spec:
  selector:
    matchLabels:
      app: bunkerweb
  template:
    metadata:
      labels:
        app: bunkerweb
    spec:
      containers:
        - name: bunkerweb-scheduler
          image: bunkerity/bunkerweb:1.6.0-rc2
          env:
            - name: KUBERNETES_MODE
              value: "yes"
            - name: DATABASE_URI
              value: "mariadb+pymysql://bunkerweb:changeme@svc-bunkerweb-db:3306/db"
            - name: DNS_RESOLVERS
              value: "coredns.kube-system.svc.cluster.local"
            - name: API_WHITELIST_IP
              value: "127.0.0.0/8 10.0.0.0/8"
            - name: MULTISITE
              value: "yes"
            - name: USE_REDIS
              value: "yes"
            - name: REDIS_HOST
              value: "svc-bunkerweb-redis.default.svc.cluster.local"

Then, enter a shell in the pod with kubectl exec -it bunkerweb-xxxxxxxxx-yyyyy -- bash. There, running env | grep REDIS shows that the variables are set correctly at the pod level, but cat /etc/nginx/variables.env | grep REDIS shows that bunkerweb misses them.

[fl0ppy-d1sk]

Hello @civts,

Your Deployment is pretty strange : it looks like the scheduler one but you use the bunkerweb image and bunkerweb names.

In 1.6.X, env variables are now defined on bunkerweb-scheduler pod as shown here : https://docs.bunkerweb.io/1.6.0-rc2/web-ui/#__tabbed_1_4

Don't hesitate to try again and tell us if it's fixed.

[civts]

Hi @fl0ppy-d1sk. That was it!

By defining the variables on the bunkerweb-scheduler, they are read correctly.
It seems to me like this is also the case for @schmittse: we both defined the REDIS_ variables in bunkerity/bunkerweb instead of bunkerity/bunkerweb-scheduler

Thank you🙏

[TheophileDiot]

Hi @civts, it is written in the documentation when upgrading from 1.5.x to 1.6.x.
https://docs.bunkerweb.io/1.6.0-rc2/upgrading/#bunkerweb-container