Feature: Need support for IRSA for spark-infrastructure helm charts

[jocobtt]

Description

Currently, the Spark-Infrastructure Helm charts constrain downstream consumers to using an IAM user for authentication to read/write from S3 buckets. This limitation restricts flexibility, compromises security, and increases the operational burden for deploying workloads in environments where IAM users may not be ideal. Instead, IAM Roles for Service Accounts (IRSA) provide a more secure and manageable alternative for authentication.

Can we potentially add support for IRSA by including the option to configure a service account in the following charts within the spark-infrastructure helm chart:

• Thrift Server chart
• Spark-History chart

Why This is Needed:

In previous versions of these charts, it was straightforward to add service account configurations by editing the manifests in the local templates folder. However, now that the charts are being consumed downstream, modifying the manifests directly is no longer feasible.To enable IRSA support, we as a downstream consumer currently rely on Kustomize patches to add the necessary service account configurations to the helm charts.

Native IRSA support in the Helm charts would simplify integration, reduce operational overhead, and align with standard Kubernetes patterns for AWS authentication.

DOD

Acceptance criteria required to realize the requested feature

• IRSA authentication mechanism supported for spark-infrastructure helm charts

Test Strategy/Script

When IRSA authentication is enabled in a project's values.yaml file with the necessary IAM role annotation, the system should:

1. Properly authenticate using the assigned IAM role.
2. Successfully read/write to S3 buckets associated with Spark-Infrastructure workloads.

References/Additional Context

• aiSSEMBLE version: 1.10.0+
• IRSA supporting documentation: AWS EKS - IAM Roles for Service Accounts