How to handle sensitive values in chart 7.1.0?

[vdmkenny]

We tried upgrading to the latest helm chart v7.1.0.
In the changelog it mentions failing if you have sensitive secrets in your values file. Currently, we do (auth.azuread).

The readme hasn't been updated, so currently It's not clear to us how to proceed.
Any documention for this?

[diranged]

I think this is related to #2899.

[jim-barber-he]

Also hit this and don't know how to proceed (except by setting assertNoLeakedSecrets: false but I wasn't aware sensitive values were in a config map so would prefer to solve the issue properly)
In our values that are expanded via helmfile at deploy time, we had the following (amongst other values I haven't shown):

grafana.ini:
  auth.google:
    client_secret: {{ env "GOOGLE_CLIENT_SECRET" }}

I tried changing this to:

env:
  GOOGLE_CLIENT_SECRET: {{ env "GOOGLE_CLIENT_SECRET" }}

grafana.ini:
  auth.google:
    client_secret: ${GOOGLE_CLIENT_SECRET}

But still get the same error.
Also tried this with no such luck:

env:
  GOOGLE_CLIENT_SECRET: {{ env "GOOGLE_CLIENT_SECRET" }}

grafana.ini:
  auth.google:
    client_secret: $__env{GOOGLE_CLIENT_SECRET}

Maybe grafana.ini should be stored in a Secret instead of a ConfigMap if it can contain sensitive values?

[aofei]

I believe the solution (also the best practice (?) for passing sensitive values to grafana.ini) is to override configuration with environment variables.

For example, replace:

grafana.ini:
  auth.google:
    client_secret: ${GOOGLE_CLIENT_SECRET}

with

envFromSecret: grafana-env

and

apiVersion: v1
kind: Secret
metadata:
  name: grafana-env
  namespace: grafana
stringData:
  GF_AUTH_GOOGLE_CLIENT_SECRET: ...

[jim-barber-he]

Thanks.
The solution above means needing to pre-create the secret yourself.
A simpler way that works for me is:

envRenderSecret:
  GF_AUTH_GOOGLE_CLIENT_SECRET: {{ env "GOOGLE_CLIENT_SECRET" }}

And remove the client_secret part from the grafana.ini config.

That way the helm chart creates the secret for me. The relevant parts of the helm diff:

                value: /var/lib/grafana/plugins
              - name: GF_PATHS_PROVISIONING
                value: /etc/grafana/provisioning
+           envFrom:
+             - secretRef:
+                 name: grafana-env
            livenessProbe:
              failureThreshold: 10
              httpGet:

prometheus, grafana-env, Secret (v1) has been added:
- 
+ # Source: grafana/templates/secret-env.yaml
+ apiVersion: v1
+ kind: Secret
+ metadata:
+   name: grafana-env
+   namespace: prometheus
+   labels:
+     helm.sh/chart: grafana-7.1.0
+     app.kubernetes.io/name: grafana
+     app.kubernetes.io/instance: grafana
+     app.kubernetes.io/version: "10.2.3"
+     app.kubernetes.io/managed-by: Helm
+ type: Opaque
+ data:
+   GF_AUTH_GOOGLE_CLIENT_SECRET: ...

[timo1707]

as mentioned here i am facing the same issue, but only within the CI/CD run.

according to the Grafana Helm Documentation the secrets should be passed via the extraSecretMounts stanza. which i tried (was using before the envFromSecrets stanza) but both ways are not working anymore.

[anarsen]

I got it working. This is the diff from the chart values file:

   values:
+    envValueFrom:
+      GF_AUTH_AZUREAD_CLIENT_SECRET:
+        secretKeyRef:
+          name: oidc
+          key: client_secret
+      GF_DATABASE_PASSWORD:
+        secretKeyRef:
+          name: database
+          key: password
+
     "grafana.ini":
       database:
         name: $__file{/etc/secrets/database/name}
         host: $__file{/etc/secrets/database/host}
         user: $__file{/etc/secrets/database/username}
-        password: "$__file{/etc/secrets/database/password}"
         ssl_mode: require
         server_cert_name: grafana
         ca_cert_path: /etc/secrets/database/ca_cert
       "auth.azuread":
         enabled: true
         name: Azure AD
         allow_signup: true
         scopes: "openid email profile"
         client_id: $__file{/etc/secrets/oidc/client_id}
-        client_secret: $__file{/etc/secrets/oidc/client_secret}
         auth_url: $__file{/etc/secrets/oidc/authorize_url}
         token_url: $__file{/etc/secrets/oidc/token_url}
         allow_assign_grafana_admin: true
         use_refresh_token: true

[P-n-I]

Maybe I'm missing something but I wouldn't expect:

client_id: $__file{/etc/secrets/auth_generic_oauth/client_id}

to fail validation. I've created a Secret with the client_secret in it and used extraSecretMounts to mount it in the relevant place. This worked in 7.0.*.

[asdutoit]

I have a very similar issue when trying to define the username and password for the SMTP server configuration.
Also using extraSecretMounts. I have confirmed the secrets are mounted within the pod but the grafana isn't picking up the secrets in its config even though it is specified within the grafana.ini file both in yaml and within the pod:

PS, I am using the kube-prometheus-stack Helm chart.

values.yaml file:

extraSecretMounts:
    - name: grafana-email-secret
      secretName: grafana-email-secret
      mountPath: /etc/secrets/grafana
      defaultMode: 0444
      readOnly: true
grafana.ini:
    smtp:
      enabled: true
      host: "email-smtp.us-west-1.amazonaws.com:587"
      existingSecret: ""
      from_address: "[email protected]"
      userKey: $__file{/etc/secrets/grafana/smtp_user}    # I tried both "user" and "userKey"
      passwordKey: $__file{/etc/secrets/grafana/smtp_password}    # I tried both "password" and "passwordKey"
      skip_verify: false

Within the Grafana POD:

kubectl exec -it -n observability pod/kube-prometheus-stack-grafana-5b7b8978c8-jn6nm -- /bin/sh      
                                                
/usr/share/grafana $ cat /etc/secrets/grafana/smtp_user
SUPERSECRETPASSWORD_1234

/usr/share/grafana $ cat /etc/grafana/grafana.ini 
[analytics]
check_for_updates = true
[grafana_net]
url = https://grafana.net
[log]
mode = console
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[server]
domain = example.com
root_url = https://grafana.example.com
[smtp]
enabled = true
existingSecret = 
from_address = [email protected]
host = email-smtp.us-west-1.amazonaws.com:587
passwordKey = $__file{/etc/secrets/grafana/smtp_password}
skip_verify = false
userKey = $__file{/etc/secrets/grafana/smtp_user}

Screenshot from Grafana UI:

It does work find when I manually specify the Username and Password, but this not the way I would prefer to work.

[bdalpe]

FWIW -- you can use the following syntax for files in environment variable configs: GF_<section>_<key>__FILE e.g. GF_SMTP_PASSWORD__FILE=/etc/secrets/grafana/smtp_password (https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables)

This is documented here: https://github.com/grafana/grafana/blob/48a5c1e8509980b6ffe8cf5c5e486d096eb7058c/packaging/docker/run.sh#L49-L61

I have a PR (#2904) open to fix the issue with "secrets" in the grafana.ini configs.

[asdutoit]

Thanks @bdalpe , I tried your suggestion and it works beautifully.
For the record, here is my config:

grafana:
  env:
    GF_SMTP_USER__FILE: /etc/secrets/grafana/smtp_user
    GF_SMTP_PASSWORD__FILE: /etc/secrets/grafana/smtp_password
  extraSecretMounts:
    - name: grafana-email-secret
      secretName: grafana-email-secret
      mountPath: /etc/secrets/grafana
      defaultMode: 0440
      readOnly: true
  grafana.ini:
    smtp:
      enabled: true
      host: "email-smtp.us-west-1.amazonaws.com:587"
      existingSecret: ""
      from_address: "[email protected]"
      skip_verify: false

[bdalpe]

This is fixed with chart version 7.2.2.

[younsl]

In my case, I configured Grafana v10.3.1 using the variable expansion feature as the error message advised.

 

TLDR

• For security purposes, Grafana v7.1+, if a plaintext secret is entered in the values.yaml file, installation will fail.
• The Variable Expansion feature allows you to hide sensitive information, such as Google OAuth Secret, in your values.yaml file.
• A simple workaround is to add the assertNoLeakedSecrets: false value to the Grafana Helm chart and force install it, but this is not recommended.

# values for kube-prometheus-stack
...
grafana:
  # assertNoLeakedSecrets is a helper function defined in _helpers.tpl that checks if secret
  # values are not exposed in the rendered grafana.ini configmap. It is enabled by default.
  #
  # To pass values into grafana.ini without exposing them in a configmap, use variable expansion:
  # https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#variable-expansion
  #
  # Alternatively, if you wish to allow secret values to be exposed in the rendered grafana.ini configmap,
  # you can disable this check by setting assertNoLeakedSecrets to false.
  assertNoLeakedSecrets: false
  ...

 

Google OAuth integration with grafana using variable expension

Problem

If the values.yaml file contains Google oauth-related secret and ID values in plain text, installation will be rejected.

helm upgrade \
  kube-prometheus-stack . \
  --install \
  --create-namespace \
  --namespace monitoring \
  --values values.yaml \
  --wait

Error: UPGRADE FAILED: execution error at (kube-prometheus-stack/charts/grafana/templates/statefulset.yaml:28:28): Sensitive key 'auth.google.client_secret' should not be defined explicitly in values. Use variable expansion instead. You can disable this client-side validation by changing the value of assertNoLeakedSecrets

This error message is triggered by a conditional statement declared in the _helpers.tpl file of the Grafana Helm chart.

_helpers.tpl file in Grafana 7.3.0 helm chart:

  {{- if $.Values.assertNoLeakedSecrets -}}
      {{- $grafanaIni := index .Values "grafana.ini" -}}
      {{- range $_, $secret := $sensitiveKeysYaml.sensitiveKeys -}}
        {{- $currentMap := $grafanaIni -}}
        {{- $shouldContinue := true -}}
        {{- range $index, $elem := $secret.path -}}
          {{- if and $shouldContinue (hasKey $currentMap $elem) -}}
            {{- if eq (len $secret.path) (add1 $index) -}}
              {{- if not (regexMatch "\\$(?:__(?:env|file|vault))?{[^}]+}" (index $currentMap $elem)) -}}
                {{- fail (printf "Sensitive key '%s' should not be defined explicitly in values. Use variable expansion instead. You can disable this client-side validation by changing the value of assertNoLeakedSecrets." (join "." $secret.path)) -}}
              {{- end -}}
            {{- else -}}
              {{- $currentMap = index $currentMap $elem -}}
            {{- end -}}
          {{- else -}}
              {{- $shouldContinue = false -}}
          {{- end -}}
        {{- end -}}
      {{- end -}}
  {{- end -}}
{{- end -}}

Environment

• EKS v1.29
• EC2 worker nodes based amd64 CPU
• helm chart
  • kube-prometheus-stack v56.6.2 (Includes grafana v10.3.1 / 00a22ff8b2)

How to setup

Create a new secret resource containing Google OAuth client id and secret.

kubectl create secret generic kube-prometheus-stack-grafana-oauth \
  --from-literal GF_AUTH_GOOGLE_CLIENT_ID="<REDACTED>" \
  --from-literal GF_AUTH_GOOGLE_CLIENT_SECRET="<REDACTED>" \
  --namespace monitoring

secret/kube-prometheus-stack-grafana-oauth created

Add grafana.ini values to the kube-prometheus-stack helm chart

Note:
Variable expansion is only available in Grafana 7.1+

# values.yaml for kube-promethue-stack
...
## Using default values from https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml
##
grafana:
  envFromSecret: kube-prometheus-stack-grafana-oauth

  grafana.ini:
    security:
      cookie_secure: false
    dashboard:
      min_refresh_interal: 60s
    auth.google:
      enabled: true
      allow_sign_up: true
      auto_login: false
      # Variable expansion is only availble in Grafana 7.1+
      # https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#variable-expansion
      client_id: $__env{GF_AUTH_GOOGLE_CLIENT_ID}
      client_secret: $__env{GF_AUTH_GOOGLE_CLIENT_SECRET}
      scopes: openid email profile
      auth_url: https://accounts.google.com/o/oauth2/v2/auth
      token_url: https://oauth2.googleapis.com/token
      api_url: https://openidconnect.googleapis.com/v1/userinfo

Install(or upgrade) kube-prometheus-stack release on your EKS cluster.

helm upgrade \
  kube-prometheus-stack . \
  --install \
  --create-namespace \
  --namespace monitoring \
  --values values.yaml \
  --wait

Release "kube-prometheus-stack" has been upgraded. Happy Helming!
NAME: kube-prometheus-stack
LAST DEPLOYED: Thu Feb 15 15:43:27 2024
NAMESPACE: monitoring
STATUS: deployed
REVISION: 9
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

The Google OAuth ID and Secret are injected into the Pod as environment variables. These environment variables are taken from the kubernetes secret we created in the previous step.

$ kubectl exec -it -n monitoring kube-prometheus-stack-grafana-0 -- env | grep GF_AUTH_GOOGLE_CLIENT_
GF_AUTH_GOOGLE_CLIENT_ID=<REDACTED>-<REDACTED>.apps.googleusercontent.com
GF_AUTH_GOOGLE_CLIENT_SECRET=<REDACTED>

[prashant0085]

Is someone able to do something similar for LDAP?

It seems bind_password of ldap needs to be in ldap.toml file and we define this file in grafana.ini. I am not able to substitute values ldap.toml. Is there a way to pass a variable in grafana.ini which can be used in ldap.toml?

[bdalpe]

@prashant0085 environment variables are supported in TOML according to the Grafana docs: https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/ldap/#using-environment-variables

bind_password = "${LDAP_ADMIN_PASSWORD}"

Then set in your Helm values:

grafana:
  env:
    LDAP_ADMIN_PASSWORD__FILE: /etc/secrets/grafana/ldap_password

I haven't tested this, but in theory it should work.

[prashant0085]

@bdalpe Thanks for quick response. I have tried both the below syntax in ldap.toml as per Grafana documentation on this page: https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/ldap/#using-environment-variables

bind_password = "${LDAP_BIND_PASSWORD}"
bind_password = '$__env{LDAP_BIND_PASSWORD}'

Also I have the bind password as kubernetes secret, which I am converting to env variable by setting envFromSecret in values.yaml

I have checked and confirmed that password is available as env variable in grafana pod. Sharing echo command output below:

monitoring-grafana-5ffccdb96c-6dnk9:~$ echo $LDAP_BIND_PASSWORD
"""password#12345"""

values.yaml below

envFromSecret: grafana-ldap-secrets

grafana.ini:
    auth.ldap:
      enabled: true
      allow_sign_up: true
      config_file: /etc/grafana/ldap.toml
      skip_org_role_sync: false 

  ldap:
    enabled: true
    config: |-
      verbose_logging = true

      [[servers]]
      host = "ad.example.com"
      port = 636
      use_ssl = true
      start_tls = false
      ssl_skip_verify = true
      bind_dn = "cn=LDAP BIND,cn=Users,dc=ad,dc=example,dc=com"
      bind_password = '$__env{LDAP_BIND_PASSWORD}' or bind_password = "${LDAP_BIND_PASSWORD}"
      search_filter = "(|(sAMAccountName=%s)(mail=%s)(cn=%s)(name=%s))"
      search_base_dns = ["ou=OML-Users,dc=ad,dc=example,dc=com"]
      group_search_filter = "(objectClass=group)"
      group_search_base_dns = ["OU=OML-Users,DC=ad,DC=example,DC=com"]
      group_search_filter_user_attribute = "distinguishedName"

      [servers.attributes]
      name = "distinguishedName"
      username = "sAMAccountName"
      email = "mail"

Instead of actual value ldap.toml is getting populated with actual variable name which I defined in values.yaml starting with $

and when I hard code the password manually in /etc/grafana/ldap.toml it works absolutely fine

bind_password = """password#12345"""

[bdalpe]

@prashant0085 You're mixing LDAP_BIND_PASSWORD and LDAP_ADMIN_PASSWORD in your examples.

Would also recommend replacing the single quotes with double quotes.

[prashant0085]

@bdalpe I took the code from example, but in my values.yaml I have always used LDAP_BIND_PASSWORD with single and double quotes both. But I will try again with all below options and let you know the outcome:

bind_password = '${LDAP_BIND_PASSWORD}'
bind_password = '$__env{LDAP_BIND_PASSWORD}'
bind_password = "${LDAP_BIND_PASSWORD}"
bind_password = "$__env{LDAP_BIND_PASSWORD}"

UPDATE: tested and it doesn't work, I am using kube-proemtheus-stack helm chart 58.6.1

[evangraudins]

I'm also unable to get the variables in grafana.ini to work.

[alonapester]

@younsl solution worked for me with kube-prometheus-stack, thanks