Error when decrypting secrets on ArgoCD

[peyerroger]

Current Behavior

Error:
Error decrypting tree: Error walking tree: Could not decrypt value: crypto/aes: invalid key size 0

Full Error

rpc error: code = Unknown desc = `helm template . --name-template app-argocd --namespace argo-cd-2 --kube-version 1.21 --values values.yaml --values secrets+gpg-import:///helm-secrets-key/key.asc?secrets.yaml --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1beta1 --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1beta1 --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1beta1 --api-versions apps/v1 --api-versions argoproj.io/v1alpha1 --api-versions authentication.k8s.io/v1 --api-versions authentication.k8s.io/v1beta1 --api-versions authorization.k8s.io/v1 --api-versions authorization.k8s.io/v1beta1 --api-versions autoscaling/v1 --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta2 --api-versions batch/v1 --api-versions batch/v1beta1 --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1beta1 --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1beta1 --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1beta1 --api-versions extensions/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions metrics.k8s.io/v1beta1 --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1beta1 --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1beta1 --api-versions policy/v1 --api-versions policy/v1beta1 --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1beta1 --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1beta1 --api-versions snapshot.storage.k8s.io/v1 --api-versions snapshot.storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1beta1 --api-versions v1 --include-crds` failed exit status 2: **Error decrypting tree: Error walking tree: Could not decrypt value: crypto/aes: invalid key size 0** panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1ae9207] goroutine 1 [running]: bytes.(*Buffer).Bytes(...) bytes/buffer.go:54 helm.sh/helm/v3/pkg/cli/values.readFile(0xc0004d2e40, 0x3d, 0xc000711500, 0x3, 0x4, 0xc000831800, 0x0, 0x0, 0x0, 0x0) helm.sh/helm/v3/pkg/cli/values/options.go:120 +0x167 helm.sh/helm/v3/pkg/cli/values.(*Options).MergeValues(0xc0007de8a0, 0xc000711500, 0x3, 0x4, 0x1, 0x47, 0x0) helm.sh/helm/v3/pkg/cli/values/options.go:48 +0x115 main.runInstall(0xc000835500, 0x1, 0x62, 0xc0006a6420, 0xc0007de8a0, 0x2292040, 0xc000130008, 0x0, 0x1ffbaea, 0x9) helm.sh/helm/v3/cmd/helm/install.go:193 +0x24e main.newTemplateCmd.func2(0xc0001df400, 0xc000835500, 0x1, 0x62, 0x0, 0x0) helm.sh/helm/v3/cmd/helm/template.go:82 +0x146 github.com/spf13/cobra.(*Command).execute(0xc0001df400, 0xc000834e00, 0x62, 0x70, 0xc0001df400, 0xc000834e00) github.com/spf13/[email protected]/command.go:852 +0x472 github.com/spf13/cobra.(*Command).ExecuteC(0xc0005faa00, 0xc00000e318, 0x1, 0xc00079ff60) github.com/spf13/[email protected]/command.go:960 +0x375 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/[email protected]/command.go:897 main.main() helm.sh/helm/v3/cmd/helm/helm.go:80 +0x25b

Config:

Repo Server

argo-cd:
  repoServer:
    env:
      - name: HELM_PLUGINS
        value: /custom-tools/helm-plugins/
      - name: HELM_SECRETS_SOPS_PATH
        value: /custom-tools/sops
      - name: HELM_SECRETS_KUBECTL_PATH
        value: /custom-tools/kubectl
    volumes:
      - name: custom-tools
        emptyDir: {}
      - name: helm-secrets-key
        secret:
          secretName: helm-secrets-key
    volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
      - mountPath: /helm-secrets-key/
        name: helm-secrets-key
    initContainers:
      - name: download-tools
        image: alpine:latest
        command: [sh, -ec]
        env:
          - name: HELM_SECRETS_VERSION
            value: "3.9.1"
          - name: SOPS_VERSION
            value: "3.7.1"
          - name: KUBECTL_VERSION
            value: "1.22.2"
        args:
          - |
            mkdir -p /custom-tools/helm-plugins
            wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

            wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
            wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

            chmod +x /custom-tools/*
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

Secret:

apiVersion: v1
kind: Secret
metadata:
  name: helm-secrets-key
data:
  key.asc: {{ .Values.helmSecretsKey | toString | b64enc }}
type: Opaque

I verified that the key is mounted. Also the tools are mounted. If I execute a similar command as on ArgoCD locally, everything is working fine.:

helm upgrade abc-argo-cd . -f values.yaml -f 'secrets+gpg-import:///Users/abc/dev/misc/keys/key.asc?secrets.yaml' --namespace=argo-cd --create-namespace --install

Helm version on Argo is 3.6.0. On my local machine it is 3.7.1. Could it be a dependency issue?

Expected Behavior

The secret should be decrypted.

Steps To Reproduce

No response

Environment

• Helm 3.6.0 on ArgoCD

      - name: HELM_SECRETS_VERSION
        value: "3.9.1"
      - name: SOPS_VERSION
        value: "3.7.1"
      - name: KUBECTL_VERSION
        value: "1.22.2"
  

Anything else?

No response

[jkroepke]

Whats your local version of gpg?

gpg --version

Just to confirm, we are not hitting getsops/sops#896

[peyerroger]

First of all, thanks a lot for the fast response!

Local: gpg (GnuPG) 2.3.3
ArgoCD: gpg (GnuPG) 2.2.20

So it really looks like that this is the issue :(

Where is the dependency coming from on the ArgoCD image? Is my assumption right, that I'd have to create my own docker image in case I want to upgrade gpg?

[jkroepke]

Where is the dependency coming from on the ArgoCD image? Is my assumption right, that I'd have to create my own docker image in case I want to upgrade gpg?

ArgoCD is using Ubuntu 21.04 as base. But eben Ubuntu 21.10 stays on gpg 2.2. You could try to use the gnupg packages from debian experimental to upgrade gpg inside the docker image.

OR

Do you have a chance to regenerate your gpg key? If yes, add --rfc4880 on key generation.

See: getsops/sops#896 (comment)

[peyerroger]

I can confirm that the workaround with the --rfc4880 is working.

Thanks a lot @jkroepke for helping out!