diff --git a/.config/functional_tests/pre-entrypoint-helpers.sh b/.config/functional_tests/pre-entrypoint-helpers.sh index a1bc78c..3daeec8 100644 --- a/.config/functional_tests/pre-entrypoint-helpers.sh +++ b/.config/functional_tests/pre-entrypoint-helpers.sh @@ -3,4 +3,8 @@ ## use this to load any configuration before the functional test ## TIPS: avoid modifying the .project_automation/functional_test/entrypoint.sh ## migrate any customization you did on entrypoint.sh to this helper script -echo "Executing Pre-Entrypoint Helpers" \ No newline at end of file +echo "Executing Pre-Entrypoint Helpers" + +#********** TFC Env Vars ************* +export AWS_DEFAULT_REGION=us-east-1 +export AWS_REGION=us-east-1 diff --git a/.gitignore b/.gitignore index 3a992ad..66bbe9c 100644 --- a/.gitignore +++ b/.gitignore @@ -43,4 +43,10 @@ go.mod go.sum # Terraform tests -tests/*.auto.tfvars \ No newline at end of file +tests/*.auto.tfvars + +*.tfplan* +**/builds* + +.DS_Store +**/.DS_Store diff --git a/.header.md b/.header.md index 5e652f5..13f04a3 100644 --- a/.header.md +++ b/.header.md @@ -1,7 +1,29 @@ -# Terraform Module Project +# Terraform Module for Amazon Redshift Copy UDF -:no_entry_sign: Do not edit this readme.md file. To learn how to change this content and work with this repository, refer to CONTRIBUTING.md +This terraform module provides complimentary capabilities to +[COPY command](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) +by enabling data copy from S3 API compliant storage solutions such as +[Cloudian](https://github.com/cloudian/cloudian-s3-operator), +[MinIO](https://github.com/minio/minio), and +[Weka](https://github.com/weka/csi-wekafs) into Amazon Redshift with +AWS Lambda UDF (User Defined Function). -## Readme Content +## Architecture Diagram -This file will contain any instructional information about this module. +![Architecture Diagram](./docs/diagram.png "Architecture Diagram") + +## Usage + +```terraform +module "udf" { + source = "aws-ia/redshift-copy-udf/aws" + version = "~> 1.0" + + name = "redshift-copy-udf" + memory_size = 128 + timeout = 5 + + vpc_subnet_ids = null # replace with comma separated values + security_group_ids = null # replace with comma separated values +} +``` diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 5b627cf..5120cc4 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,4 +1,5 @@ -## Code of Conduct +# Code of Conduct + This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact -opensource-codeofconduct@amazon.com with any additional questions or comments. +`opensource-codeofconduct@amazon.com` with any additional questions or comments. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2c113ca..15cdce2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,7 +8,7 @@ For best practices and information on developing with Terraform, see the [I&A Mo ## Contributing Code -In order to contibute code to this repository, you must submit a *[Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)*. To do so, you must *[fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo)* this repostiory, make your changes in your forked version and submit a *Pull Request*. +In order to contribute code to this repository, you must submit a *[Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)*. To do so, you must *[fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo)* this repository, make your changes in your forked version and submit a *Pull Request*. ## Writing Documentation @@ -20,7 +20,7 @@ README.md is automatically generated by pulling in content from other files. For Pull Requests (PRs) submitted against this repository undergo a series of static and functional checks. -> :exclamation: Note: Failures during funtional or static checks will prevent a pull request from being accepted. +> :exclamation: Note: Failures during functional or static checks will prevent a pull request from being accepted. It is a best practice to perform these checks locally prior to submitting a pull request. @@ -37,7 +37,7 @@ TIPS: **do not** modify the `./project_automation/{test-name}/entrypoint.sh`, in - Checkov - Terratest -> :bangbang: The readme.md file will be created after all checks have completed successfuly, it is recommended that you install terraform-docs locally in order to preview your readme.md file prior to publication. +> :bangbang: The readme.md file will be created after all checks have completed successfully, it is recommended that you install terraform-docs locally in order to preview your readme.md file prior to publication. ## Install the required tools diff --git a/LICENSE b/LICENSE index 261eeb9..e70b68f 100644 --- a/LICENSE +++ b/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index 19effa5..53eb31b 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,87 @@ -# Terraform Module Project +# Terraform Module for Amazon Redshift Copy UDF -:no_entry_sign: Do not edit this readme.md file. To learn how to change this content and work with this repository, refer to CONTRIBUTING.md +This terraform module provides complimentary capabilities to +[COPY command](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html) +by enabling data copy from S3 API compliant storage solutions such as +[Cloudian](https://github.com/cloudian/cloudian-s3-operator), +[MinIO](https://github.com/minio/minio), and +[Weka](https://github.com/weka/csi-wekafs) into Amazon Redshift with +AWS Lambda UDF (User Defined Function). -## Readme Content +## Architecture Diagram -This file will contain any instructional information about this module. +![Architecture Diagram](./docs/diagram.png "Architecture Diagram") + +## Usage + +```terraform +module "udf" { + source = "aws-ia/redshift-copy-udf/aws" + version = "~> 1.0" + + name = "redshift-copy-udf" + memory_size = 128 + timeout = 5 + + vpc_subnet_ids = null # replace with comma separated values + security_group_ids = null # replace with comma separated values +} +``` ## Requirements +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.0.0 | +| [aws](#requirement\_aws) | >= 4.0.0 | +| [random](#requirement\_random) | >= 3.0.0 | + ## Providers -No providers. +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 4.0.0 | +| [random](#provider\_random) | >= 3.0.0 | ## Modules -No modules. +| Name | Source | Version | +|------|--------|---------| +| [lambda](#module\_lambda) | terraform-aws-modules/lambda/aws | ~> 7.0 | ## Resources -No resources. +| Name | Type | +|------|------| +| [aws_iam_role.redshift](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy.redshift](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [random_id.this](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource | +| [aws_partition.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source | +| [aws_secretsmanager_secret.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret) | data source | +| [aws_secretsmanager_secret_version.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret_version) | data source | ## Inputs -No inputs. +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [memory\_size](#input\_memory\_size) | Lambda UDF memory size | `number` | `128` | no | +| [name](#input\_name) | Lambda UDF function name | `string` | `"redshift-copy-udf"` | no | +| [security\_group\_ids](#input\_security\_group\_ids) | Security Group IDs (comma separated values) | `string` | `null` | no | +| [storage\_pass](#input\_storage\_pass) | Storage Password to Access S3 API Compliant Storage | `string` | `null` | no | +| [storage\_secret\_arn](#input\_storage\_secret\_arn) | Secrets Manager ARN for S3 API Compliant Storage Credentials | `string` | `null` | no | +| [storage\_url](#input\_storage\_url) | Storage URL to Access S3 API Compliant Storage | `string` | `null` | no | +| [storage\_user](#input\_storage\_user) | Storage Username to Access S3 API Compliant Storage | `string` | `null` | no | +| [timeout](#input\_timeout) | Lambda UDF timeout | `number` | `300` | no | +| [vpc\_subnet\_ids](#input\_vpc\_subnet\_ids) | VPC Subnet IDs (comma separated values) | `string` | `null` | no | ## Outputs -No outputs. - +| Name | Description | +|------|-------------| +| [iam\_role\_arn](#output\_iam\_role\_arn) | IAM Role ARN for Redshift Permissions | +| [iam\_role\_id](#output\_iam\_role\_id) | IAM Role ID for Redshift Permissions | +| [iam\_role\_name](#output\_iam\_role\_name) | IAM Role Name for Redshift Permissions | +| [lambda\_function\_arn](#output\_lambda\_function\_arn) | Lambda Function ARN for Redshift UDF | +| [lambda\_function\_name](#output\_lambda\_function\_name) | Lambda Function Name for Redshift UDF | + \ No newline at end of file diff --git a/VERSION b/VERSION index ae39fab..0ec25f7 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v0.0.0 +v1.0.0 diff --git a/data.tf b/data.tf new file mode 100644 index 0000000..6a73259 --- /dev/null +++ b/data.tf @@ -0,0 +1,14 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +data "aws_partition" "this" {} + +data "aws_secretsmanager_secret" "this" { + count = local.secret_count + arn = var.storage_secret_arn +} + +data "aws_secretsmanager_secret_version" "this" { + count = local.secret_count + secret_id = data.aws_secretsmanager_secret.this[0].id +} diff --git a/docs/diagram.png b/docs/diagram.png new file mode 100644 index 0000000..1e0e87a Binary files /dev/null and b/docs/diagram.png differ diff --git a/examples/basic/README.md b/examples/basic/README.md index f53c234..af8e74e 100644 --- a/examples/basic/README.md +++ b/examples/basic/README.md @@ -3,21 +3,29 @@ | Name | Version | |------|---------| -| [terraform](#requirement\_terraform) | >= 0.14.0 | -| [aws](#requirement\_aws) | >= 3.72.0 | -| [awscc](#requirement\_awscc) | >= 0.11.0 | +| [terraform](#requirement\_terraform) | >= 1.0.0 | +| [aws](#requirement\_aws) | >= 4.0.0 | ## Providers -No providers. +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 4.0.0 | ## Modules -No modules. +| Name | Source | Version | +|------|--------|---------| +| [udf](#module\_udf) | ../../ | n/a | ## Resources -No resources. +| Name | Type | +|------|------| +| [aws_availability_zones.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source | +| [aws_caller_identity.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source | +| [aws_partition.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source | +| [aws_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source | ## Inputs @@ -25,5 +33,11 @@ No inputs. ## Outputs -No outputs. +| Name | Description | +|------|-------------| +| [iam\_role\_arn](#output\_iam\_role\_arn) | IAM Role ARN for Redshift Permissions | +| [iam\_role\_id](#output\_iam\_role\_id) | IAM Role ID for Redshift Permissions | +| [iam\_role\_name](#output\_iam\_role\_name) | IAM Role Name for Redshift Permissions | +| [lambda\_function\_arn](#output\_lambda\_function\_arn) | Lambda Function ARN for Redshift UDF | +| [lambda\_function\_name](#output\_lambda\_function\_name) | Lambda Function Name for Redshift UDF | \ No newline at end of file diff --git a/examples/basic/data.tf b/examples/basic/data.tf new file mode 100644 index 0000000..332e9bf --- /dev/null +++ b/examples/basic/data.tf @@ -0,0 +1,7 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +data "aws_region" "this" {} +data "aws_partition" "this" {} +data "aws_caller_identity" "this" {} +data "aws_availability_zones" "this" {} diff --git a/examples/basic/main.tf b/examples/basic/main.tf index b2619ce..1ecc4fb 100644 --- a/examples/basic/main.tf +++ b/examples/basic/main.tf @@ -1,5 +1,15 @@ -##################################################################################### -# Terraform module examples are meant to show an _example_ on how to use a module -# per use-case. The code below should not be copied directly but referenced in order -# to build your own root module that invokes this module -##################################################################################### +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +module "udf" { + source = "../../" + # source = "aws-ia/redshift-copy-udf/aws" + # version = "~> 1.0" + + name = "redshift-copy-udf" + memory_size = 128 + timeout = 5 + + vpc_subnet_ids = null # replace with comma separated values + security_group_ids = null # replace with comma separated values +} diff --git a/examples/basic/outputs.tf b/examples/basic/outputs.tf index e69de29..9b819b5 100644 --- a/examples/basic/outputs.tf +++ b/examples/basic/outputs.tf @@ -0,0 +1,27 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +output "iam_role_arn" { + description = "IAM Role ARN for Redshift Permissions" + value = module.udf.iam_role_arn +} + +output "iam_role_id" { + description = "IAM Role ID for Redshift Permissions" + value = module.udf.iam_role_id +} + +output "iam_role_name" { + description = "IAM Role Name for Redshift Permissions" + value = module.udf.iam_role_name +} + +output "lambda_function_arn" { + description = "Lambda Function ARN for Redshift UDF" + value = module.udf.lambda_function_arn +} + +output "lambda_function_name" { + description = "Lambda Function Name for Redshift UDF" + value = module.udf.lambda_function_name +} diff --git a/examples/basic/providers.tf b/examples/basic/providers.tf index 0f413cb..409c022 100644 --- a/examples/basic/providers.tf +++ b/examples/basic/providers.tf @@ -1,21 +1,12 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + terraform { - required_version = ">= 0.14.0" + required_version = ">= 1.0.0" required_providers { aws = { source = "hashicorp/aws" - version = ">= 3.72.0" - } - awscc = { - source = "hashicorp/awscc" - version = ">= 0.11.0" + version = ">= 4.0.0" } } } - -provider "awscc" { - user_agent = [{ - product_name = "terraform-awscc-" - product_version = "0.0.1" - comment = "V1/AWS-D69B4015/" - }] -} diff --git a/examples/minio/.header.md b/examples/minio/.header.md new file mode 100644 index 0000000..e69de29 diff --git a/examples/minio/README.md b/examples/minio/README.md new file mode 100644 index 0000000..42383a5 --- /dev/null +++ b/examples/minio/README.md @@ -0,0 +1,60 @@ + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.0.0 | +| [aws](#requirement\_aws) | >= 4.0.0 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 4.0.0 | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [eks](#module\_eks) | terraform-aws-modules/eks/aws | ~> 20.0 | +| [eks\_mng](#module\_eks\_mng) | terraform-aws-modules/eks/aws//modules/eks-managed-node-group | ~> 20.0 | +| [udf](#module\_udf) | ../../ | n/a | +| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 | +| [vpc\_endpoints](#module\_vpc\_endpoints) | terraform-aws-modules/vpc/aws//modules/vpc-endpoints | ~> 5.0 | + +## Resources + +| Name | Type | +|------|------| +| [aws_eks_addon.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon) | resource | +| [aws_iam_policy.eks](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | +| [aws_iam_role.redshift](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy_attachment.redshift](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_kms_alias.alias](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_alias) | resource | +| [aws_kms_key.kms](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource | +| [aws_redshiftserverless_namespace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/redshiftserverless_namespace) | resource | +| [aws_redshiftserverless_workgroup.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/redshiftserverless_workgroup) | resource | +| [aws_availability_zones.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source | +| [aws_caller_identity.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source | +| [aws_partition.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source | +| [aws_region.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [cidr](#input\_cidr) | This is the CIDR block for your EKS cluster | `string` | `"10.0.0.0/16"` | no | +| [k8s](#input\_k8s) | This is the version of your EKS cluster | `string` | `"1.29"` | no | +| [name](#input\_name) | This is the name of your EKS cluster | `string` | `"redshift-minio-demo"` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [iam\_role\_arn](#output\_iam\_role\_arn) | IAM Role ARN for Redshift Permissions | +| [iam\_role\_id](#output\_iam\_role\_id) | IAM Role ID for Redshift Permissions | +| [iam\_role\_name](#output\_iam\_role\_name) | IAM Role Name for Redshift Permissions | +| [lambda\_function\_arn](#output\_lambda\_function\_arn) | Lambda Function ARN for Redshift UDF | +| [lambda\_function\_name](#output\_lambda\_function\_name) | Lambda Function Name for Redshift UDF | +| [storage\_instructions](#output\_storage\_instructions) | n/a | + \ No newline at end of file diff --git a/examples/minio/data.tf b/examples/minio/data.tf new file mode 100644 index 0000000..332e9bf --- /dev/null +++ b/examples/minio/data.tf @@ -0,0 +1,7 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +data "aws_region" "this" {} +data "aws_partition" "this" {} +data "aws_caller_identity" "this" {} +data "aws_availability_zones" "this" {} diff --git a/examples/minio/locals.tf b/examples/minio/locals.tf new file mode 100644 index 0000000..3e27fd6 --- /dev/null +++ b/examples/minio/locals.tf @@ -0,0 +1,51 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +locals { + azs = slice(data.aws_availability_zones.this.names, 0, 3) + addons = ["aws-ebs-csi-driver", "eks-pod-identity-agent", "kube-proxy", "vpc-cni"] + arn_prefix = format("arn:%s", data.aws_partition.this.partition) + arn_suffix = format("%s:%s", data.aws_region.this.name, data.aws_caller_identity.this.account_id) + arn_eks_cluster = format("%s:%s:%s:cluster/%s", local.arn_prefix, "eks", local.arn_suffix, var.name) + arn_eks_nodegroup = format("%s:%s:%s:nodegroup/%s/*/*", local.arn_prefix, "eks", local.arn_suffix, var.name) + arn_redshift = format("%s:iam::aws:policy/AmazonRedshiftAllCommandsFullAccess", local.arn_prefix) + + redshift_params = [ + { + parameter_key = "auto_mv" + parameter_value = "true" + }, + { + parameter_key = "datestyle" + parameter_value = "ISO, MDY" + }, + { + parameter_key = "enable_case_sensitive_identifier" + parameter_value = "false" + }, + { + parameter_key = "enable_user_activity_logging" + parameter_value = "true" + }, + { + parameter_key = "max_query_execution_time" + parameter_value = "14400" + }, + { + parameter_key = "query_group" + parameter_value = "default" + }, + { + parameter_key = "require_ssl" + parameter_value = "false" + }, + { + parameter_key = "search_path" + parameter_value = "$user, public" + }, + { + parameter_key = "use_fips_ssl" + parameter_value = "false" + } + ] +} diff --git a/examples/minio/main.tf b/examples/minio/main.tf new file mode 100644 index 0000000..93acf8f --- /dev/null +++ b/examples/minio/main.tf @@ -0,0 +1,260 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +####################### +# Redshift UDF Module # +####################### +module "udf" { + source = "../../" + # source = "aws-ia/redshift-copy-udf/aws" + # version = "~> 1.0" + memory_size = 128 + timeout = 5 + vpc_subnet_ids = join(",", module.vpc.private_subnets) + security_group_ids = module.vpc.default_security_group_id +} + +################## +# VPC Constructs # +################## +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "~> 5.0" + name = "${var.name}-vpc" + azs = local.azs + cidr = var.cidr + private_subnets = [for k, v in local.azs : cidrsubnet(var.cidr, 4, k)] + public_subnets = [for k, v in local.azs : cidrsubnet(var.cidr, 8, k + 48)] + enable_nat_gateway = true + single_nat_gateway = true + + default_security_group_name = "${var.name}-sg" + default_security_group_egress = [{ + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = "0.0.0.0/0" + }] + default_security_group_ingress = [{ + from_port = 0 + to_port = 0 + protocol = "-1" + self = true + }, { + from_port = 443 + to_port = 443 + protocol = "tcp" + cidr_blocks = "0.0.0.0/0" + }] +} + +module "vpc_endpoints" { + source = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints" + version = "~> 5.0" + vpc_id = module.vpc.vpc_id + + endpoints = { + dynamodb = { + service = "dynamodb" + service_type = "Gateway" + route_table_ids = flatten([ + module.vpc.intra_route_table_ids, + module.vpc.private_route_table_ids, + module.vpc.public_route_table_ids + ]) + }, + s3 = { + service = "s3" + service_type = "Gateway" + route_table_ids = flatten([ + module.vpc.intra_route_table_ids, + module.vpc.private_route_table_ids, + module.vpc.public_route_table_ids + ]) + }, + } +} + +################## +# EKS Constructs # +################## +module "eks" { + source = "terraform-aws-modules/eks/aws" + version = "~> 20.0" + cluster_name = var.name + cluster_version = var.k8s + cluster_endpoint_public_access = true + + vpc_id = module.vpc.vpc_id + subnet_ids = module.vpc.private_subnets + control_plane_subnet_ids = module.vpc.private_subnets + enable_cluster_creator_admin_permissions = true + + node_security_group_tags = { + "kubernetes.io/cluster/${var.name}" = null + } +} + +module "eks_mng" { + source = "terraform-aws-modules/eks/aws//modules/eks-managed-node-group" + version = "~> 20.0" + name = var.name + cluster_name = var.name + cluster_version = var.k8s + cluster_service_cidr = module.eks.cluster_service_cidr + + cluster_primary_security_group_id = module.eks.cluster_primary_security_group_id + vpc_security_group_ids = [module.eks.node_security_group_id, module.vpc.default_security_group_id] + subnet_ids = module.vpc.private_subnets + + ami_type = "AL2023_x86_64_STANDARD" + capacity_type = "SPOT" + instance_types = ["t3.medium"] + min_size = 0 + max_size = 2 + desired_size = 1 + disk_size = 50 +} + +resource "aws_eks_addon" "this" { + count = length(local.addons) + cluster_name = var.name + addon_name = element(local.addons, count.index) + depends_on = [module.eks_mng] +} + +resource "aws_iam_policy" "eks" { + name = "${var.name}-eks-policy" + path = "/" + + policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow", + "Action" : [ + "eks:DescribeNodegroup", + "eks:ListNodegroups", + "eks:UpdateNodegroupConfig" + ], + "Resource" : [ + "${local.arn_eks_cluster}", + "${local.arn_eks_nodegroup}" + ] + }, + { + "Effect" : "Allow", + "Action" : [ + "eks:CreateNodegroup", + "eks:TagResource", + "iam:PassRole", + "iam:ListAttachedRolePolicies" + ], + "Resource" : "${local.arn_eks_cluster}" + }, + { + "Effect" : "Allow", + "Action" : [ + "autoscaling:Describe*", + "ec2:RunInstances", + "ec2:DescribeSubnets", + "ec2:DescribeLaunchTemplateVersions", + "ec2:CreateTags", + "iam:GetRole" + ], + "Resource" : "*" + }, + { + "Effect" : "Allow", + "Action" : [ + "aws-marketplace:MeterUsage", + "aws-marketplace:ResolveCustomer", + "aws-marketplace:BatchMeterUsage", + "aws-marketplace:GetEntitlements", + "aws-marketplace:RegisterUsage" + ], + "Resource" : "*" + } + ] + }) +} + +####################### +# Redshift Constructs # +####################### +resource "aws_kms_key" "kms" { + deletion_window_in_days = 7 + enable_key_rotation = true + key_usage = "ENCRYPT_DECRYPT" + + policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow" + "Action" : "kms:*" + "Resource" : "*" + "Principal" : { + "AWS" : data.aws_caller_identity.this.account_id + } + } + ] + }) +} + +resource "aws_kms_alias" "alias" { + name = "alias/redshift-serverless" + target_key_id = aws_kms_key.kms.key_id +} + +resource "aws_redshiftserverless_namespace" "this" { + namespace_name = "${var.name}-namespace" + kms_key_id = aws_kms_key.kms.arn + iam_roles = [aws_iam_role.redshift.arn] + log_exports = ["connectionlog", "useractivitylog", "userlog"] + manage_admin_password = true +} + +resource "aws_redshiftserverless_workgroup" "this" { + namespace_name = aws_redshiftserverless_namespace.this.namespace_name + workgroup_name = "${var.name}-workgroup" + enhanced_vpc_routing = true + publicly_accessible = false + security_group_ids = [module.vpc.default_security_group_id] + subnet_ids = module.vpc.private_subnets + + dynamic "config_parameter" { + for_each = local.redshift_params + content { + parameter_key = config_parameter.value.parameter_key + parameter_value = config_parameter.value.parameter_value + } + } +} + +resource "aws_iam_role" "redshift" { + name = "${var.name}-redshift-role" + path = "/service-role/" + + assume_role_policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow" + "Action" : "sts:AssumeRole" + "Principal" : { + "Service" : [ + "redshift.amazonaws.com", + "redshift-serverless.amazonaws.com", + "sagemaker.amazonaws.com" + ] + } + } + ] + }) +} + +resource "aws_iam_role_policy_attachment" "redshift" { + role = aws_iam_role.redshift.name + policy_arn = local.arn_redshift +} diff --git a/examples/minio/minio-tenant.yaml b/examples/minio/minio-tenant.yaml new file mode 100644 index 0000000..e06ef6e --- /dev/null +++ b/examples/minio/minio-tenant.yaml @@ -0,0 +1,49 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: Namespace +metadata: + name: minio-tenant +--- +apiVersion: v1 +kind: Secret +metadata: + name: storage-configuration + namespace: minio-tenant +type: Opaque +stringData: + config.env: |- + export MINIO_ROOT_USER="minio" + export MINIO_ROOT_PASSWORD="minio123" + export MINIO_STORAGE_CLASS_STANDARD="EC:2" + export MINIO_BROWSER="on" +--- +apiVersion: minio.min.io/v2 +kind: Tenant +metadata: + name: minio + namespace: minio-tenant +spec: + serviceMetadata: + minioServiceAnnotations: + service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0a4cecff434c80e6e" + exposeServices: + console: true + minio: true + configuration: + name: storage-configuration + image: quay.io/minio/minio:RELEASE.2023-05-27T05-56-19Z + pools: + - servers: 4 + name: pool-0 + volumesPerServer: 4 + volumeClaimTemplate: + apiVersion: v1 + kind: PersistentVolumeClaim + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 16Gi diff --git a/examples/minio/outputs.tf b/examples/minio/outputs.tf new file mode 100644 index 0000000..283342c --- /dev/null +++ b/examples/minio/outputs.tf @@ -0,0 +1,52 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +output "iam_role_arn" { + description = "IAM Role ARN for Redshift Permissions" + value = module.udf.iam_role_arn +} + +output "iam_role_id" { + description = "IAM Role ID for Redshift Permissions" + value = module.udf.iam_role_id +} + +output "iam_role_name" { + description = "IAM Role Name for Redshift Permissions" + value = module.udf.iam_role_name +} + +output "lambda_function_arn" { + description = "Lambda Function ARN for Redshift UDF" + value = module.udf.lambda_function_arn +} + +output "lambda_function_name" { + description = "Lambda Function Name for Redshift UDF" + value = module.udf.lambda_function_name +} + +output "storage_instructions" { + value = < len(lines): + print(f"[ERROR] line number out of range: {nr_line} > {len(lines)}") + raise MyException("line number out of range") + + rec = [] + if nr_line < 0: + rec.append(str(len(lines))) + elif lines: + for line in lines[nr_line:nr_line + nr_rec]: + rec.append(line) + + if len(rec) != nr_rec: + print(f"[ERROR] number of records mismatch: {len(rec)} != {nr_rec}") + raise MyException("number of records mismatch") + + result["success"] = True + result["results"] = rec + + except MyException as e: + result["error_msg"] = str(e) + print(f"[ERROR] {str(e)}") + + except Exception as e: + result["error_msg"] = str(e) + print(f"[ERROR] {str(e)}") + exc_type, exc_obj, exc_tb = sys.exc_info() + print(f"[ERROR] exc_info: {exc_type}, {exc_tb.tb_lineno}") + + return json.dumps(result) diff --git a/lambda_cfn/output.csv b/lambda_cfn/output.csv new file mode 100644 index 0000000..e69de29 diff --git a/lambda_udf/function.py b/lambda_udf/function.py new file mode 100644 index 0000000..f1eecbd --- /dev/null +++ b/lambda_udf/function.py @@ -0,0 +1,86 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import boto3, json, sys, os +from urllib import parse + +kwargs = {} +if "STORAGE_URL" in os.environ and os.environ["STORAGE_URL"]: + kwargs["endpoint_url"] = os.environ["STORAGE_URL"] + kwargs["verify"] = False +if "STORAGE_USER" in os.environ and os.environ["STORAGE_USER"]: + kwargs["aws_access_key_id"] = os.environ["STORAGE_USER"] +if "STORAGE_PASS" in os.environ and os.environ["STORAGE_PASS"]: + kwargs["aws_secret_access_key"] = os.environ["STORAGE_PASS"] +if "STORAGE_TOKEN" in os.environ and os.environ["STORAGE_TOKEN"]: + kwargs["aws_session_token"] = os.environ["STORAGE_TOKEN"] + +client = boto3.client('s3', **kwargs) +cached = {} + +class MyException(Exception): + """Raise for my exceptions""" + +def handler(event, context): + print(f"[DEBUG] event: {event}") + nr_rec = int(event["num_records"]) if "num_records" in event else 1 + result = {"success": False, "num_records": nr_rec} + + try: + if "arguments" not in event or not event["arguments"]: + print("[ERROR] event arguments are missing") + raise MyException("event arguments are missing") + + if not event["arguments"][0]: + print(f"[ERROR] s3 url is missing") + raise MyException("s3 url is missing") + + if len(event["arguments"][0]) < 2: + print(f"[ERROR] line number is missing") + raise MyException("line number is missing") + + nr_line = int(event["arguments"][0][1]) + url = parse.urlparse(event["arguments"][0][0]) + print(f"[DEBUG] url parse: {url}") + + key = abs(hash(url.netloc + url.path)) % (10 ** 8) + if key in cached: + lines = cached[key] + print(f"[DEBUG] cached lines count: {len(lines)}") + else: + obj = client.get_object(Bucket=url.netloc, Key=url.path.lstrip("/")) + print(f"[DEBUG] s3 object: {obj}") + content = obj['Body'].read().decode('utf-8') + lines = content.splitlines() + cached[key] = lines + print(f"[DEBUG] uncached lines count: {len(lines)}") + + if nr_line > len(lines): + print(f"[ERROR] line number out of range: {nr_line} > {len(lines)}") + raise MyException("line number out of range") + + rec = [] + if nr_line < 0: + rec.append(str(len(lines))) + elif lines: + for line in lines[nr_line:nr_line + nr_rec]: + rec.append(line) + + if len(rec) != nr_rec: + print(f"[ERROR] number of records mismatch: {len(rec)} != {nr_rec}") + raise MyException("number of records mismatch") + + result["success"] = True + result["results"] = rec + + except MyException as e: + result["error_msg"] = str(e) + print(f"[ERROR] {str(e)}") + + except Exception as e: + result["error_msg"] = str(e) + print(f"[ERROR] {str(e)}") + exc_type, exc_obj, exc_tb = sys.exc_info() + print(f"[ERROR] exc_info: {exc_type}, {exc_tb.tb_lineno}") + + return json.dumps(result) diff --git a/lambda_udf/function.sql b/lambda_udf/function.sql new file mode 100644 index 0000000..15e901d --- /dev/null +++ b/lambda_udf/function.sql @@ -0,0 +1,16 @@ +--- # Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +--- # SPDX-License-Identifier: Apache-2.0 + +-- /* +-- Purpose: +-- This sample function demonstrates how to create/use Lambda UDF(s) in Redshift +-- 2024-09-26: written by eistrati +-- */ + +CREATE OR REPLACE EXTERNAL FUNCTION f_redshift_copy_udf (s3Url varchar, lineNumber integer) +RETURNS varchar(65534) STABLE +LAMBDA 'redshift-copy-udf' IAM_ROLE ':RedshiftRole'; + +-- SELECT +-- generate_series(0, 100) AS id, +-- f_redshift_copy_udf('s3://{{bucket_name}}/{{file_name}}.csv', id); diff --git a/locals.tf b/locals.tf new file mode 100644 index 0000000..cb14028 --- /dev/null +++ b/locals.tf @@ -0,0 +1,19 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +locals { + iam_policies_arns = [ + "arn:${data.aws_partition.this.partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole", + "arn:${data.aws_partition.this.partition}:iam::aws:policy/AmazonS3ReadOnlyAccess", + ] + secret_count = try(trimspace(var.storage_secret_arn), "") != "" ? 1 : 0 + env_vars = ( + local.secret_count > 0 + ? jsondecode(data.aws_secretsmanager_secret_version.this[0].secret_string) + : { + STORAGE_URL = var.storage_url + STORAGE_USER = var.storage_user + STORAGE_PASS = var.storage_pass + } + ) +} diff --git a/main.tf b/main.tf index e69de29..a025015 100644 --- a/main.tf +++ b/main.tf @@ -0,0 +1,85 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +############################## +# Lambda Function Constructs # +############################## +resource "random_id" "this" { + byte_length = 4 + + keepers = { + spf_gid = format("%s-%s-%s", var.name, var.memory_size, var.timeout) + } +} + +module "lambda" { + source = "terraform-aws-modules/lambda/aws" + version = "~> 7.0" + function_name = format("%s-%s", var.name, random_id.this.hex) + handler = "function.handler" + runtime = "python3.9" + memory_size = var.memory_size + timeout = var.timeout + source_path = "${path.module}/lambda_udf" + + create_role = true + attach_policies = true + number_of_policies = length(local.iam_policies_arns) + policies = local.iam_policies_arns + role_path = "/service-role/" + policy_path = "/service-role/" + + environment_variables = { + for key, value in local.env_vars : + key => value if try(trimspace(value), "") != "" + } + vpc_security_group_ids = ( + try(trimspace(var.security_group_ids), "") != "" + ? split(",", var.security_group_ids) : null + ) + vpc_subnet_ids = ( + try(trimspace(var.vpc_subnet_ids), "") != "" + ? split(",", var.vpc_subnet_ids) : null + ) +} + +#################################### +# IAM Role for Redshift Constructs # +#################################### +resource "aws_iam_role" "redshift" { + name = format("%s-%s-role", var.name, random_id.this.hex) + path = "/service-role/" + + assume_role_policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow" + "Action" : "sts:AssumeRole" + "Principal" : { + "Service" : [ + "redshift.amazonaws.com", + "redshift-serverless.amazonaws.com", + "sagemaker.amazonaws.com" + ] + } + } + ] + }) +} + +resource "aws_iam_role_policy" "redshift" { + name = format("%s-%s-policy", var.name, random_id.this.hex) + role = aws_iam_role.redshift.id + + policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow", + "Action" : "lambda:InvokeFunction", + "Resource" : module.lambda.lambda_function_arn + } + ] + }) +} diff --git a/outputs.tf b/outputs.tf index e69de29..7abf4ba 100644 --- a/outputs.tf +++ b/outputs.tf @@ -0,0 +1,27 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +output "iam_role_arn" { + description = "IAM Role ARN for Redshift Permissions" + value = aws_iam_role.redshift.arn +} + +output "iam_role_id" { + description = "IAM Role ID for Redshift Permissions" + value = aws_iam_role.redshift.id +} + +output "iam_role_name" { + description = "IAM Role Name for Redshift Permissions" + value = aws_iam_role.redshift.name +} + +output "lambda_function_arn" { + description = "Lambda Function ARN for Redshift UDF" + value = module.lambda.lambda_function_arn +} + +output "lambda_function_name" { + description = "Lambda Function Name for Redshift UDF" + value = module.lambda.lambda_function_name +} diff --git a/providers.tf b/providers.tf index e492a2c..18d8f07 100644 --- a/providers.tf +++ b/providers.tf @@ -1,13 +1,16 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + terraform { - required_version = ">= 1.0.7" + required_version = ">= 1.0.0" required_providers { aws = { source = "hashicorp/aws" - version = ">= 4.0.0, < 5.0.0" + version = ">= 4.0.0" } - awscc = { - source = "hashicorp/awscc" - version = ">= 0.24.0" + random = { + source = "hashicorp/random" + version = ">= 3.0.0" } } } diff --git a/variables.tf b/variables.tf index e69de29..cc92b31 100644 --- a/variables.tf +++ b/variables.tf @@ -0,0 +1,56 @@ +# Copyright (C) Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +variable "name" { + type = string + default = "redshift-copy-udf" + description = "Lambda UDF function name" +} + +variable "memory_size" { + type = number + default = 128 + description = "Lambda UDF memory size" +} + +variable "timeout" { + type = number + default = 300 + description = "Lambda UDF timeout" +} + +variable "security_group_ids" { + type = string + default = null + description = "Security Group IDs (comma separated values)" +} + +variable "vpc_subnet_ids" { + type = string + default = null + description = "VPC Subnet IDs (comma separated values)" +} + +variable "storage_secret_arn" { + type = string + default = null + description = "Secrets Manager ARN for S3 API Compliant Storage Credentials" +} + +variable "storage_url" { + type = string + default = null + description = "Storage URL to Access S3 API Compliant Storage" +} + +variable "storage_user" { + type = string + default = null + description = "Storage Username to Access S3 API Compliant Storage" +} + +variable "storage_pass" { + type = string + default = null + description = "Storage Password to Access S3 API Compliant Storage" +}