External Secrets not working on Fargate

[csantanapr]

Description

When using the external-secrets addon and the addon runs on Fargate pods, the validation webhook deployment pod doesn't work, the api-server can't connect with errors

The root cause is that the helm chart for external-secrets uses the port 10250 and when the pod runs in fargate it conflict with kubelet port 10250

We should update

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Versions

• Module version [Required]: latest released

• Terraform version: latest

• Provider version(s): latest

Reproduction Code [Required]

Steps to reproduce the behavior:

• Deploy cluster with Fargate, create fargate profile to run external-secrets on it
• Enable external-dns module

Expected behaviour

external-dns to be ok

Actual behaviour

error

Terminal Output Screenshot(s)

erros on api-server:

Error: cluster-secretstore-sm failed to run apply: error when creating "/tmp/138385594kubectl_manifest.yaml": Internal error occurred: failed calling webhook "[validate.clustersecretstore.external-secrets.io](http://validate.clustersecretstore.external-secrets.io/)": failed to call webhook: Post "[https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clustersecretstore?timeout=5s](https://external-secrets-webhook.external-secrets.svc/validate-external-secrets-io-v1beta1-clustersecretstore?timeout=5s)": x509: certificate is valid for ip-10-0-1-115.ec2.internal, not external-secrets-webhook.external-secrets.svc (edited)

Additional context

An issue was opened to see if the helm chart default value port could be change for new releases
https://github.com/external-secrets/external-secrets/issues/19815

Workaround for now to set the port to 9443 and make sure security group rules allows access from control plane to nodes on this port.

  # Enable External Secrets Operator
  enable_external_secrets = true
  external_secrets_helm_config = {  
    namespace = "external-secrets",
    values = [
      yamlencode(
        {
      "webhook" : {
          port" = "9443"
        }
      }
    ]
  }

We should default to port 9443 for the external-dns module.

This was found during customer POC

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[bryantbiggs]

the EKS module provides this access by default which should resolve this issue under v5 now https://github.com/terraform-aws-modules/terraform-aws-eks/blob/df1b62548c1d8c7117f4ab45c5b494de64b34cb8/node_groups.tf#L165-L172

[bryantbiggs]

sorry - mis-read, looks like we need to set addon to use this port.

[fcarta29]

Correct PR link is external-secrets/external-secrets#1981

[TinoSM]

Got this issue, still lost 1-2hours trying to find the solution until I found this, confirmed as working in my end aswell (I changed the port for my system in helm values)

[alanwu4321]

latest workaround on fargate deployment for me is

enable_external_secrets = true

# ISSUES: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/issues/55
external_secrets = {
  values = [
    yamlencode({ "webhook" : { "port" = "9443" } })
  ]
}