Unable to decrypt files via sops and KMS using v0.26.1 version

[groenator]

I am running Flux using Openshift 4.8 platform. Flux is installed via the Openshift Operator community marketplace.

However, since (last week) I installed the latest version of the operator (v0.26.1) I am no longer able to decrypt files from our repository using the AWS credentials set for flux.

When I create the Kustomization resource, I am getting the below error;

oc get ks
NAME READY STATUS AGE
mgmt-dep False decryption failed for 'cloud-creds': DataWithFormat: GetDataKey: Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
arn:aws:kms:ID*: FAILED
- | Error decrypting key: NoCredentialProviders: no valid
| providers in chain. Deprecated.
| For verbose messaging see
| aws.Config.CredentialsChainVerboseErrors

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were. 11m

The cloud-creds is the secret where the AWS credentials are stored.

The AWS credentials for sops are deployed using a credentials file inside the kustomize-controller pod located under the /home/controller/.aws folder.

I tested the creds inside the cluster, making sure that there aren't any issues with the creds or networking. No issue was found there either, I was able to encrypt/decrypt the files manually.

Here is what my configuration looks like:

AWS:
Credential file with its own profile:

[iam]
aws_access_key_id=ACCESS_KEY_ID
aws_secret_access_key=SECRET_ACCESS_KEY

SOPS:
.sops.yaml file:

creation_rules:
- path_regex: cloudbuild_encrypted.yaml$
  kms: arn:aws:kms:ID*
  aws_profile: iam
- path_regex: encrypted\.yaml$
  kms: arn:aws:kms:ID*
  encrypted_regex: ^(data|stringData)$
  aws_profile: iam

I forgot to mention I also tested kustomize to load the AWS credentials directly from a k8s secret. The same issue as above where sops is not reading the credentials file anymore.

I am using the same setup on others clusters without any issues, with a lower version of flux. I can confirm using the 0.25.3 flux version works fine.

What changed recently? I am aware that sops is managed by another team, can you confirm what version of sops is running inside flux?

Did sops make a change in their API where is not able to read the credentials files anymore?

Thank you for your time and I hope the information provided is enough.

[stefanprodan]

AWS requires an fsGroup to be specified for non root containers, but Openshift doesn't like that. You need to restore the fsGroup and use SCC, all of this is on OperatorHub readme: https://operatorhub.io/operator/flux

[groenator]

Thanks @stefanprodan

I will look into this!

Regards,

[groenator]

Hi @stefanprodan

I am re-opening this issue because I think the issue I am encountering now its different.

I tried the above steps, non of the steps describe in the docs worked and the issue is still occuring.

What I am confused about it is the error messege.

How does Flux knows the secret name when the kustomization resoure is configured to mount a volume from a secret and dump the secret into .aws folder? Is not configure to read the secret directly from k8s secrets.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: mgmt-dep
  namespace: github
spec:
  decryption:
    provider: sops
  force: false
  interval: 10m
  path: <path to folder>
  prune: false
  sourceRef:
    kind: GitRepository
    name: master
  timeout: 2m

Thank you!

[stefanprodan]

How does Flux knows the secret name when the kustomization resoure is configured to mount a volume from a secret and dump the secret into .aws folder?

Flux uses the AWS SDK, the SDK knows where the token is located.

Have you restarted kustomize-controller after adding the IAM role binding to the kustomize-controller service account? For AWS to mount the secret inside the pod, you need to delete the controller pod after you create the role binding.

[groenator]

Hi @stefanprodan

It looks that the cloud-creds is the first file that sops is trying to decrypt. It has nothing to do with the secret. It happens that both names are the same.

I did remove flux, add the permissions and then install flux again via the Subscription model.

I will look into this more and keep you update it.

Regards,

[groenator]

Hi,

Setting the env in the Openshift operator subscription to:

    env:
    - name: AWS_PROFILE
      value: iam-local
    - name: HOME
      value: /home/{$USER}

Would fix this issue. The last version of the operator is changing the $HOME path to /, sops is not able to decrypt the secrets because is not reading the AWS creds profile correctly.

Thank you guys for your help.

This issue can be closed.

Regards,

[stefanprodan]

@groenator thanks for posting the solution for Openshift.

@aryan9600 we need to add this to the kustomize-controller docs in fluxcd/kustomize-controller#641

[groenator]

@stefanprodan you are welcome!