move terraform-aws-eks-blueprints-addon inside terraform-aws-eks-blueprints-addons

[schwichti]

• ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

Cannot we move terraform-aws-eks-blueprints-addon inside terraform-aws-eks-blueprints-addons?
I just need a few addons, but all needs to be downloaded when I run terraform init, because all submodules reference source = "aws-ia/eks-blueprints-addon/aws".

Provide a link to the example/module related to the question

Additional context

[bryantbiggs]

I'm sorry, I don't quite follow - what is the exact issue that you are encountering?

[schwichti]

Lets say I just want to use aws_efs_csi_driver and ingress_nginx:

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0" 
  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn
  enable_aws_efs_csi_driver = true
  enable_ingress_nginx      = true
}

When I run terraform init it will download all the other add-ons, e.g., argo_events, even though I do not need them.
In addition, they contain the same terraform files as they share the same source aws-ia/eks-blueprints-addon/aws.

Am I using the module correctly? Is there currently a way to avoid unnecessary downloads?

A possible solution might be to move the terraform-aws-eks-blueprints-addon repo into the terraform-aws-eks-blueprints-addons repo as a submodule and then reference with a relative path. This way at least terraform-aws-eks-blueprints-addon has not to be downloaded multiple times. Still not ideal, but maybe someone has a better idea.

[FernandoMiguel]

you can use the source of the module you want, avoiding to download all of them.
we do this for the half dozen we use

[bryantbiggs]

terraform-aws-eks-blueprints-addon (singular) is separate for a reason - its the basic building block which allows users/partners to deploy a Helm release with or without IRSA. This is the externalized form of what used to be irsa and helm-addon

This distinction and separation is important to avoid issues created in the past

[schwichti]

you can use the source of the module you want, avoiding to download all of them. we do this for the half dozen we use

@FernandoMiguel, do you mean you are partially replicating what terraform-aws-eks-blueprints-addons (plural) does inside your own Terraform files? So lets say you use aws_efs_csi_driver. In this case would you replicate (copy)

terraform-aws-eks-blueprints-addons/main.tf

Line 372 in b8b7a4f

################################################################################

until line 548 with all required locals and data resources? If I get you right, I was also considering this approach. Indeed, you won't need to download unnecessary modules, but this way you would not get any upstream updates.

[schwichti]

terraform-aws-eks-blueprints-addon (singular) is separate for a reason - its the basic building block which allows users/partners to deploy a Helm release with or without IRSA. This is the externalized form of what used to be irsa and helm-addon

This distinction and separation is important to avoid issues created in the past

@bryantbiggs We have been also affected by aws-ia/terraform-aws-eks-blueprints#1630 (comment) as you can see here dspace-group/simphera-reference-architecture-aws#89. But if terraform-aws-eks-blueprints-addon (singular) would be moved into terraform-aws-eks-blueprints-addons (plural) repo, could not partners still use it like this:

module "eks_addons" {
 source = "terraform-aws-modules/eks/aws//module/terraform-aws-eks-blueprints-addon"
version = "1.1.0"
}

[bryantbiggs]

The modules are designed from a perspective of stability and clear versioning requirements. They are not designed from the perspective of how many copies are downloaded

[FernandoMiguel]

Like this

module "eks_blueprints_kubernetes_addons_karpenter" {
  source = "git::ssh://[email protected]/aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons/karpenter?ref=v4.32.1"

This gets you karpenter only, at whatever version you want.
Singular module to download

[schwichti]

@FernandoMiguel this approach won't work with terraform-aws-eks-blueprints-addons or terraform-aws-eks-blueprints in version >=5

[FernandoMiguel]

I haven't looked in detail, but care to let me know why not?
It's a standard approach in terraform.

[schwichti]

The modules are designed from a perspective of stability and clear versioning requirements. They are not designed from the perspective of how many copies are downloaded

@bryantbiggs are those design requirements contradictory?

[schwichti]

I haven't looked in detail, but care to let me know why not? It's a standard approach in terraform.

@FernandoMiguel there are upcoming breaking changes in terraform-aws-eks-blueprints. The project is going to be restructured. However, the approach I outlined in #221 (comment) will work.

[bryantbiggs]

are those design requirements contradictory?

No

[schwichti]

@bryantbiggs How about packaging and releasing the individual addons as self-contained modules on the registry? Like so:

# aws-ia/terraform-aws-eks-blueprints-addons/blob/main/main.tf
module "aws_efs_csi_driver" {
  source  = "aws-ia/eks-blueprints-addon/aws/aws_efs_csi_driver"
  version = "x.y.z"
}

# aws-ia/eks-blueprints-addon/aws/aws_efs_csi_driver
module "base" {
  source  = "aws-ia/eks-blueprints-addon/aws"
  version = "x.y.z"
}

If you want to keep everything in one repo, I think it is feasible to write automation scripts that "cut out" the individual modules from the mono-repo and publish them to the Terraform registry automatically.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

Issue closed due to inactivity.