secret internalerror, error: data values must be of type string

[stefanfritsch]

Describe the bug

Some Kustomizations show the above error for sops-encrypted secrets, e.g.:

Secret/pgadmin/pgadmin-auth-proxy-secret internalerror, error: data values must be of type string

while

1. the exact same secret works fine for other kustomizations. I.e. to test it I just copied a working encrypted secret file from a different Kustomization to the non-working Kustomization and that didn't work either.
2. There is another working sops-encrypted secret in the same Kustomization

Works:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: grafana-auth-proxy
  namespace: flux-system
spec:
  interval: 27m13s
  path: ./manifests/grafana-auth-proxy
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cattle-monitoring-system
namePrefix: grafana-
resources:
  - grafana-auth-proxy-secret.yaml
  - grafana-auth-proxy.yaml

Doesn't:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: pgadmin
  namespace: flux-system
spec:
  interval: 27m13s
  path: ./manifests/pgadmin/overlays/pgadmin
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
components:
  - ../../components/common-metadata/
resources:
  - ../../../../templates/pgadmin/  # <- This contains a sops-encrypted secret that works
  - pgadmin-auth-proxy-secret.yaml

Steps to reproduce

I don't know why it sometimes works so I can't say how to reproduce it. It happens to two of my Kustomizations and in both cases there are other secrets in the Kustomization that work. Both are multi-level Kustomizations but, again, there are other multi-level Kustomizations that work.

In all likelihood I did something really stupid but I can't figure out what's different about those two Kustomizations and I don't know how to debug what the Kustomization controller is doing.

Expected behavior

A secret that works in one Kustomization works in the other as well.

Screenshots and recordings

No response

OS / Distro

Ubuntu 20.04.3 LTS

Flux version

flux: v0.20.1

Flux check

► checking prerequisites
✗ flux 0.20.1 <0.23.0 (new version is available, please upgrade)
✔ Kubernetes 1.20.11 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.12.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.16.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.13.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.16.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.18.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.17.1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Please update to latest Flux and see of the error message changes to fluxcd/kustomize-controller#483

[stefanfritsch]

Thanks for the quick reply!

It couldn't have been SOPS not working at all (as it didn't affect all secrets) and in fact it had nothing to do with flux. The rancher-webhook certificate had expired which led to issues with new resources. This was the error message:

Internal error occurred: failed calling webhook "rancher.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": x509: certificate has expired or is not yet valid: current time 2021-11-17T13:19:12Z is after 2021-11-13T13:04:54Z

So flux was innocent, its error message was just misleading. :)

Best regards,
Stefan