helm-operator is throwing the errors when switching the EKS EC2-instances Metadata-Service-Version from IMDSv1 to IMDSv2.

[raviranjithkumar]

Describe the bug

helm-operator is throwing the errors when switching the EKS EC2-instances Metadata-Service-Version from IMDSv1 to IMDSv2. We are using S3 bucket as a helm chart repo, after moving from IMDSv1 to IMDSv2 the S3 bucket is not reachable.

Switching from IMDSv1 to IMDSv2 is necessary because of the Security issues that we have with IMDSv1

In our Org, we use flux in production and there is a security concern to update to IMDSv2 but the current Flux is not supporting it so this fix is critical for us. We need the community help to address this issue.

I have already raised the issue for flux as well: fluxcd/flux#3384

To Reproduce

Steps to reproduce the behaviour:

1. Disable IMDSv1 and start using IMDSv2 in EC2 instances of the EKS cluster

Expected behavior

Helm-Operator should be able to connect to S3 buckets and download charts for the deployments

Logs

fetch from s3: fetch object from s3: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Additional context

• Helm Operator version: 1.2.0
• Kubernetes version: v1.15.11-eks-065dce
• Git provider: Github
• Helm repository provider: AWS-S3-bucket
• Flux version: 1.20.2

[squaremo]

Are you using a Helm plugin to get charts from S3?

[raviranjithkumar]

Are you using a Helm plugin to get charts from S3?

Yes. We are using helm-s3 plugin https://github.com/hypnoglow/helm-s3

[squaremo]

OK -- I think that should be the place to begin troubleshooting then -- there's no AWS-specific code in this operator, so far as I know.

Also, ICYMI: this operator is deprecated, because we can't keep maintaining it while also supporting Helm v2. Please consider migrating to the Flux v2 helm-controller, which (I believe) supports S3 directly.

[raviranjithkumar]

OK -- I think that should be the place to begin troubleshooting then -- there's no AWS-specific code in this operator, so far as I know.

Also, ICYMI: this operator is deprecated, because we can't keep maintaining it while also supporting Helm v2. Please consider migrating to the Flux v2 helm-controller, which (I believe) supports S3 directly.

Wondering, if there is no AWS specific code for helm operator how its authenticating to S3 bucket for downloading charts. Is the credentials being shared by flux to helm-operator which is collected through ec2metadata? If yes, it can be handled if we fix fluxcd/flux#3384.

Yes. we are thinking of migrating to Flux2 if we have the support for ECR.

[squaremo]

Is the credentials being shared by flux to helm-operator which is collected through ec2metadata?

No, the Helm operator and fluxd don't share credentials.

As it says here https://docs.fluxcd.io/projects/helm-operator/en/latest/references/chart/#use-helm-downloader-plugins, you may need to mount credentials into the controller pod. This is what helm-s3 says it needs: https://github.com/hypnoglow/helm-s3#note-on-aws-authentication. I don't know what you had to do to make it work before switching to IMDSv2, but I suspect that will need to change. I don't know enough about AWS IAM or that plugin to be able to guess any better than that, sorry.

[kingdonb]

Sorry if your issue remains unresolved. The Helm Operator is in maintenance mode, we recommend everybody upgrades to Flux v2 and Helm Controller.

A new release of Helm Operator is out this week, 1.4.4.

We will continue to support Helm Operator in maintenance mode for an indefinite period of time, and eventually archive this repository.

Please be aware that Flux v2 has a vibrant and active developer community who are actively working through minor releases and delivering new features on the way to General Availability for Flux v2.

In the mean time, this repo will still be monitored, but support is basically limited to migration issues only. I will have to close many issues today without reading them all in detail because of time constraints. If your issue is very important, you are welcome to reopen it, but due to staleness of all issues at this point a new report is more likely to be in order. Please open another issue if you have unresolved problems that prevent your migration in the appropriate Flux v2 repo.

Helm Operator releases will continue as possible for a limited time, as a courtesy for those who still cannot migrate yet, but these are strongly not recommended for ongoing production use as our strict adherence to semver backward compatibility guarantees limit many dependencies and we can only upgrade them so far without breaking compatibility. So there are likely known CVEs that cannot be resolved.

We recommend upgrading to Flux v2 which is actively maintained ASAP.

I am going to go ahead and close every issue at once today,
Thanks for participating in Helm Operator and Flux! 💚 💙