Grafana Upgrade Failed

[bgatestmg]

Grafana Helm Chart 7.1.0
2024-01-10T17:33:47.568445292Z Error: UPGRADE FAILED: template: grafana/templates/secret.yaml:1:62: executing "grafana/templates/secret.yaml" at <.Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE>: can't evaluate field GF_SECURITY_ADMIN_PASSWORD__FILE in type interface {}

We use OAuth for authentication and do not use local admin/access accounts

admin:
existingSecret: ''
passwordKey: admin-password
userKey: admin-user
adminUser: admin
affinity: {}
alerting: {}
assertNoLeakedSecrets: false
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 5
targetCPU: "60"
targetMemory: ""
behavior: {}
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
createConfigmap: true
dashboardProviders: {}
dashboards: {}
dashboardsConfigMaps: {}
datasources: {}
deploymentStrategy:
type: RollingUpdate
dnsConfig: {}
dnsPolicy: null
downloadDashboards:
env: {}
envFromSecret: ''
envValueFrom: {}
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
downloadDashboardsImage:
pullPolicy: IfNotPresent
registry: docker.io
repository: curlimages/curl
sha: ''
tag: 7.85.0
enableKubeBackwardCompatibility: false
enableServiceLinks: true
env:

• name: HTTP_HOST
  value: "0.0.0.0"
• name: POD_IP
  valueFrom:
  fieldRef:
  fieldPath: status.podIP
• name: GRAFANA_DB_PASSWORD
  valueFrom:
  configMapKeyRef:
  name: grafana-config
  key: GRAFANA_DB_PASSWORD
• name: CLIENT_ID
  valueFrom:
  configMapKeyRef:
  name: grafana-config
  key: CLIENT_ID
• name: CLIENT_SECRET
  valueFrom:
  configMapKeyRef:
  name: grafana-config
  key: CLIENT_SECRET
  envFromConfigMaps: {}
  envFromSecret: ''
  envFromSecrets: []
  envRenderSecret: {}
  envValueFrom: {}
  extraConfigmapMounts: []
  extraContainerVolumes: []
  extraContainers: ''
  extraEmptyDirMounts: []
  extraExposePorts: []
  extraInitContainers: []
  extraLabels: {}
  extraObjects: []
  extraSecretMounts: []
  extraVolumeMounts: []
  extraVolumes: []
  global:
  imagePullSecrets: []
  imageRegistry: null
  cattle:
  systemProjectId: p-dkrnd
  gossipPortName: gossip
  grafana.ini:
  analytics:
  check_for_updates: true
  grafana_net:
  url: https://grafana.net
  log:
  mode: console
  paths:
  data: /var/lib/grafana/
  logs: /var/log/grafana
  plugins: /var/lib/grafana/plugins
  provisioning: /etc/grafana/provisioning
  server:
  domain: grafana.thetmg.com
  enforce_domain: true
  root_url: https://grafana.thetmg.com
  auth.anonymous:
  enabled: true
  org_name: Tarbell Management Group
  org_role: Admin
  auth.azuread:
  allow_assign_grafana_admin: true
  allow_sign_up: true
  auth_url: >-
  https://login.microsoftonline.com/1fbd3e81-9940-428f-8ba6-463db4dcdf62/oauth2/v2.0/authorize
  auto_login: true
  client_id: ${CLIENT_ID}
  client_secret: ${CLIENT_SECRET}
  enabled: true
  role_attribute_strict: false
  scopes: openid email profile
  skip_org_role_sync: false
  token_url: >-
  https://login.microsoftonline.com/1fbd3e81-9940-428f-8ba6-463db4dcdf62/oauth2/v2.0/token
  use_pkce: true
  database:
  host: mysql-1699562096.mysql.svc.cluster.local:3306
  name: grafana
  password: ${GRAFANA_DB_PASSWORD}
  type: mysql
  user: grafana
  feature_toggles:
  correlations: true
  dashgpt: true
  extraThemes: true
  nestedFolders: true
  headlessService: false
  hostAliases: []
  image:
  pullPolicy: IfNotPresent
  pullSecrets: []
  registry: docker.io
  repository: grafana/grafana-enterprise
  sha: ''
  tag: latest
  imageRenderer:
  affinity: {}
  autoscaling:
  behavior: {}
  enabled: false
  maxReplicas: 5
  minReplicas: 1
  targetCPU: '60'
  targetMemory: ''
  containerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
  drop:
  - ALL
  readOnlyRootFilesystem: true
  seccompProfile:
  type: RuntimeDefault
  deploymentStrategy: {}
  enabled: false
  env: {}
  envValueFrom: {}
  grafanaProtocol: http
  grafanaSubPath: ''
  hostAliases: []
  image:
  pullPolicy: Always
  registry: docker.io
  repository: grafana/grafana-image-renderer
  sha: ''
  tag: latest
  networkPolicy:
  extraIngressSelectors: []
  limitEgress: false
  limitIngress: true
  nodeSelector: {}
  podAnnotations: {}
  podPortName: http
  priorityClassName: ''
  replicas: {}
  resources: {}
  revisionHistoryLimit: 10
  securityContext: {}
  service:
  appProtocol: ''
  enabled: true
  port: 8081
  portName: http
  targetPort: 8081
  serviceAccountName: ''
  serviceMonitor:
  enabled: false
  interval: 1m
  labels: {}
  path: /metrics
  relabelings: []
  scheme: http
  scrapeTimeout: 30s
  targetLabels: []
  tlsConfig: {}
  tolerations: []
  ingress:
  annotations: {}
  enabled: true
  extraPaths: []
  hosts:
  • grafana.thetmg.com
    labels: {}
    path: /
    pathType: Prefix
    tls:
  • hosts:
    • grafana.thetmg.com
      secretName: thetmg
      ingressClassName: nginx
      initChownData:
      enabled: true
      image:
      pullPolicy: IfNotPresent
      registry: docker.io
      repository: library/busybox
      sha: ''
      tag: 1.31.1
      resources: {}
      securityContext:
      capabilities:
      add:
    • CHOWN
      runAsNonRoot: false
      runAsUser: 0
      seccompProfile:
      type: RuntimeDefault
      ldap:
      config: ''
      enabled: false
      existingSecret: ''
      lifecycleHooks: {}
      livenessProbe:
      failureThreshold: 10
      httpGet:
      path: /api/health
      port: 3000
      initialDelaySeconds: 60
      timeoutSeconds: 30
      namespaceOverride: ''
      networkPolicy:
      allowExternal: true
      egress:
      blockDNSResolution: false
      enabled: false
      ports: []
      to: []
      enabled: false
      explicitNamespacesSelector: {}
      ingress: true
      nodeSelector: {}
      notifiers: {}
      persistence:
      accessModes:
  • ReadWriteOnce
    enabled: false
    extraPvcLabels: {}
    finalizers:
  • kubernetes.io/pvc-protection
    inMemory:
    enabled: false
    size: 10Gi
    type: pvc
    plugins:
• agenty-flowcharting-panel
• grafana-clock-panel
• flant-statusmap-panel
• influxdata-flightsql-datasource
• marcusolsson-json-datasource
  podDisruptionBudget:
  enabled: false
  minAvailable: 2
  podPortName: grafana
  rbac:
  create: true
  extraClusterRoleRules: []
  extraRoleRules: []
  namespaced: false
  pspEnabled: false
  pspUseAppArmor: false
  readinessProbe:
  httpGet:
  path: /api/health
  port: 3000
  replicas: {}
  resources:
  limits:
  cpu: 250m
  memory: 256Mi
  requests:
  cpu: 100m
  memory: 128Mi
  revisionHistoryLimit: 10
  securityContext:
  fsGroup: 472
  runAsGroup: 472
  runAsNonRoot: true
  runAsUser: 472
  service:
  annotations: {}
  appProtocol: ''
  enabled: true
  labels: {}
  port: 80
  portName: service
  targetPort: 3000
  type: ClusterIP
  serviceAccount:
  autoMount: true
  create: true
  labels: {}
  name: null
  nameTest: null
  serviceMonitor:
  enabled: false
  interval: 30s
  labels: {}
  metricRelabelings: []
  path: /metrics
  relabelings: []
  scheme: http
  scrapeTimeout: 30s
  targetLabels: []
  tlsConfig: {}
  sidecar:
  alerts:
  enabled: false
  env: {}
  extraMounts: []
  initAlerts: false
  label: grafana_alert
  labelValue: ''
  reloadURL: https://grafana.thetmg.com/api/admin/provisioning/alerting/reload
  resource: both
  script: null
  searchNamespace: null
  sizeLimit: {}
  skipReload: false
  watchMethod: WATCH
  dashboards:
  SCProvider: true
  defaultFolderName: null
  enabled: false
  env: {}
  extraMounts: []
  folder: /tmp/dashboards
  folderAnnotation: null
  label: grafana_dashboard
  labelValue: ''
  provider:
  allowUiUpdates: false
  disableDelete: false
  folder: ''
  foldersFromFilesStructure: false
  name: sidecarProvider
  orgid: 1
  type: file
  reloadURL: http://localhost:3000/api/admin/provisioning/dashboards/reload
  resource: both
  script: null
  searchNamespace: null
  sizeLimit: {}
  skipReload: false
  watchMethod: WATCH
  datasources:
  enabled: false
  env: {}
  initDatasources: false
  label: grafana_datasource
  labelValue: ''
  reloadURL: http://localhost:3000/api/admin/provisioning/datasources/reload
  resource: both
  script: null
  searchNamespace: null
  sizeLimit: {}
  skipReload: false
  watchMethod: WATCH
  enableUniqueFilenames: false
  image:
  registry: quay.io
  repository: kiwigrid/k8s-sidecar
  sha: ''
  tag: 1.25.2
  imagePullPolicy: IfNotPresent
  livenessProbe: {}
  notifiers:
  enabled: false
  env: {}
  initNotifiers: false
  label: grafana_notifier
  labelValue: ''
  reloadURL: http://localhost:3000/api/admin/provisioning/notifications/reload
  resource: both
  script: null
  searchNamespace: null
  sizeLimit: {}
  skipReload: false
  watchMethod: WATCH
  plugins:
  enabled: false
  env: {}
  initPlugins: false
  label: grafana_plugin
  labelValue: ''
  reloadURL: http://localhost:3000/api/admin/provisioning/plugins/reload
  resource: both
  script: null
  searchNamespace: null
  sizeLimit: {}
  skipReload: false
  watchMethod: WATCH
  readinessProbe: {}
  resources: {}
  securityContext:
  allowPrivilegeEscalation: false
  capabilities:
  drop:
  - ALL
  seccompProfile:
  type: RuntimeDefault
  smtp:
  existingSecret: ''
  passwordKey: password
  userKey: user
  testFramework:
  enabled: true
  image:
  registry: docker.io
  repository: bats/bats
  tag: v1.4.1
  imagePullPolicy: IfNotPresent
  securityContext: {}
  tolerations: []
  topologySpreadConstraints: []
  useStatefulSet: false

[diranged]

We're doing the sane thing of passing in variable references to secrets, and it's erroring out as well:

    [auth.okta]...
    client_id = $__file{/etc/grafana/okta-oauth/clientID}
    client_secret = $__file{/etc/grafana/okta-oauth/clientSecret}

[aofei]

I believe the solution is #2896 (comment).

[bgatestmg]

i will give this a try now: #2896 (comment) i will report here

[bgatestmg]

Okay so i have now created a secret containing my MYSQL Database Username/Password encoded in Base64, and am injecting envFromSecret: grafana-env
auth.azuread:
client_id: ${CLIENT_ID}
client_secret: ${CLIENT_SECRET}

database:
host: mysql-1699562096.mysql.svc.cluster.local:3306
name: grafana
password: ${MYSQL_DB_PW}
type: mysql
user: grafana

[bgatestmg]

So if i deploy it manually it does indeed work without issue, however when i try to deploy with CI/CD(i am using Fleet) i get an error:
As a test i also tried encapsulating the variables in quotes and no dice.

[timo1707]

@bgatestmg same for me.
when deploying the chart with my local maschine its working, but on my Git Runner i get the similiar error

Error: UPGRADE FAILED: execution error at (grafana/templates/deployment.yaml:36:28): Sensitive key 'database.password' should not be defined explicitly in values. Use variable expansion instead. Error: Process completed with exit code 1.

i was running my .yaml with the envFromSecret stanza and changed for debugging to the extraSecretMounts stanza as in the official Documentation is described.
Both ways are working locally but end in an error with Ci/CI

[bgatestmg]

No matter what i try to do i still get the database.password error.

I have tried to pass it along as a variable every single way i can think of. i had to revert to chart version: 7.0.22 for now

This was my most recent attempt:
extraSecretMounts:

• name: grafana-env-mount
  secretName: grafana-env
  defaultMode: 0440
  mountPath: /etc/secrets/grafana_env
  readOnly: true

password: $__file{/etc/secrets/grafana_env/MYSQL_DB_PW}

[bgatestmg]

i receive the same error even with extra secret mounts with CI/CD and locally.

[bgatestmg]

@timo1707 this was my latest attempt this morning
assertNoLeakedSecrets: true

extraSecretMounts:

• name: grafana-env-mount
  secretName: grafana-env
  defaultMode: 0440
  mountPath: /etc/secrets/grafana_env
  readOnly: true

  database:
  host: mysql-1699562096.mysql.svc.cluster.local:3306
  name: grafana
  password: $__file{/etc/secrets/grafana_env/MYSQL_DB_PW}
  type: mysql
  user: grafana

helm install --generate-name=true --namespace=grafana --timeout=10m0s --values=/home/shell/helm/values-grafana-7.1.0.yaml --version=7.1.0 --wait=true /home/shell/helm/grafana-7.1.0.tgz
2024-01-12T13:48:19.914654132Z Error: INSTALLATION FAILED: execution error at (grafana/templates/deployment.yaml:36:28): Sensitive key 'database.password' should not be defined explicitly in values. Use variable expansion instead.

[tijmenvandenbrink]

I'm experiencing a similar issue with the 7.2.0 release of the chart. I'm deploying the chart with Flux (using SOPS to securely store password) and followed the instructions here. It used to work perfectly until 7.0.22.

My guess is that this commit is likely the culprit. Could it be that it doesn't correctly evaluates the variable expansion? After all, I'm just using this in the values.yaml:

      auth.generic_oauth:
        ....
        client_id: $__file{/etc/secrets/auth_generic_oauth/client_id}
        client_secret: $__file{/etc/secrets/auth_generic_oauth/client_secret}

[timo1707]

temporary solution is:
assertNoLeakedSecrets: false
in the values.yaml from your helm chart.
but as @tijmenvandenbrink mentioned this commit need a fix.
@DerekTBrown : can you pls check that?

[batazor]

It's just an astounding validation solution. I'm also using argocd + sops for secure secret management, and now I get an error - due to using auth.github.client_secret

I don't mind variable chart side validation, but this validation doesn't work correctly and breaks deploys of new versions

[bgatestmg]

@batazor can we upgrade to chart version 7.2.1 now and have everything functioning properly again? or are we still waiting for the fix to be merged.

[szymonrychu]

Hello!
I was simply using another method of supplying database.password to the application and my the value looked like so for me:

grafana:
  grafana.ini:
    database:
      type: postgres
      host: grafana-postgresql:5432
      user: grafana
      password: "${POSTGRESQL_PASSWORD}"
      ssl_mode: require
  envValueFrom:
    POSTGRESQL_PASSWORD:
      secretKeyRef:
        name: grafana.grafana-postgresql.credentials.postgresql.acid.zalan.do
        key: password

I've relied on grafana itself loading environment variable in it's config file. The env (as in example above) was provided by mechanisms supported by kubernetes and helm chart itself. It worked fine until recent changes involving secrets validation.

I'm using helmfile to apply charts into my cluster. At the same time the secret is automatically generated in runtime by postgresql operator and I can't load it during chart templating (or it's not simple- it would involve getting value from k8s secrets back to CICD just to bypass this validation).

[bdalpe]

@szymonrychu what version of the chart are you using? This issue was fixed in 7.2.2

[szymonrychu]

Hello!
thank you for the answer @bdalpe - I'm using week old kube-prometheus-stack version (55.11.0).
It includes grafana in 7.2.* version, which might include the problematic upgrade.

I guess it's already resolved in further updates. Will check later on, for now I've disable leak-protection.

[bdalpe]

@szymonrychu It looks like kube-prometheus-stack version 55.11.0 was release before the fix was merged, so the bundled version of the grafana chart is 7.2.1. You'd need to upgrade your kube-prometheus-stack version to 56.0.0 to get the fixed version (56.0.0 includes grafana chart 7.2.3).

A quick way to check the bundled version:

VERSION=55.11.0; curl -sL "https://github.com/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-${VERSION}/kube-prometheus-stack-${VERSION}.tgz" | tar zxvf - -O kube-prometheus-stack/Chart.lock | grep -A2 "name: grafana"

VERSION=56.0.0; curl -sL "https://github.com/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-${VERSION}/kube-prometheus-stack-${VERSION}.tgz" | tar zxvf - -O kube-prometheus-stack/Chart.lock | grep -A2 "name: grafana"

with output:

x kube-prometheus-stack/Chart.lock
- name: grafana
  repository: https://grafana.github.io/helm-charts
  version: 7.2.1

x kube-prometheus-stack/Chart.lock
- name: grafana
  repository: https://grafana.github.io/helm-charts
  version: 7.2.3

[xyfleet]

Hi @bdalpe Why do I still encounter this error? I tried to upgrade from 55.11.0 to 56.0.0.

execution error at (kube-prometheus-stack/charts/grafana/templates/deployment.yaml:36:28): Sensitive key 'database.password' should not be defined explicitly in values. Use variable expansion instead. You can disable this client-side validation by changing the value of assertNoLeakedSecrets.

In order to fix this error, I have to configure

assertNoLeakedSecrets: false

After upgrading to 56.0.0, I tried to upgrade to 56.21.4 without the setting of assertNoLeakedSecrets and got the error again.

Any issue still here?