Amazon Managed Service for Apache Flink Application Lifecycle Management with Terraform

This code sample demonstrates how to use Terraform to control the lifecycle of a Managed Service for Apache Flink application using Docker. The process ensures consistent deployment environments and secure handling of AWS credentials.

Architecture overview

Pre-requisites

• An AWS account
• Java 11 or later
• Apache Maven 3.9.6 or later
• Docker installed and running on your machine
• AWS CLI installed and configured with appropriate AWS profile

The Terraform code in this example only manages the Managed Service for Apache Flink application. Additional resources are required, in the same account and region where you are going to deploy the application.

These resources must be manually created:

• Kinesis data stream
  • Data stream name: ExampleOutputStream
  • Capacity mode: Provisioned
  • Number of provisioned shards: 1 or more
• S3 bucket
  • Stores the application JAR. For simplicity, the same bucket is used to store the Terraform remote state. It is recommended to enable Bucket Versioning on the S3 bucket to allow for state recovery.
• DynamoDB table
  • Used by Terraform for locking. The primary key must be called LockID.

Do not forget to edit the Terraform state backend configuration as described later with the bucket name and table name.

Environment variables

Before starting, ensure you have the following environment variable set:

• AWS_PROFILE: Your AWS profile name that contains the necessary credentials

AWS Region

This example is designed to be deployed in us-east-1. Your AWS profile should match this region. If you choose to use a different region, do not forget to amend the region in the following files:

• terraform/config.tvars.json
• terraform/backend.conf
• terraform/variables.tf

Getting started

1. Clone the repository

Clone the repository to your desired workspace and move into the repository:

git clone https://github.com/aws-samples/amazon-managed-service-for-apache-flink-lifecycle-management-terraform.git

2. Build the Docker image

Build the Docker image by running the following command:

docker build -t msf-terraform .

This command builds a Docker image named msf-terraform using the Dockerfile in the current directory. The image contains all necessary dependencies for running Terraform and AWS operations.

3. Export the AWS credentials

Make sure to put your correct AWS profile name in $AWS_PROFILE and that your profile is authenticated and has sufficient permissions.

Then run the following command to extract the current credentials:

aws configure export-credentials --profile $AWS_PROFILE --format env-no-export > .env.docker

This step:

• Exports AWS credentials from your specified AWS profile
• Saves the temporary credentials in environment variable format to .env.docker that will be used by the Docker container

This workflow ensures secure credential handling without exposing them in the command line.

Important: If you authenticate with AWS using temporary credentials, often with corporate SSO, you may need to execute this command again when credentials are renewed.

4. Terraform state backend

Amazon S3 is used to store the Terraform state and Amazon DynamoDB for state locking and consistency checking.

Edit the file ./terraform/backend.conf:

bucket = "your-s3-bucket-name"
key = "terraform/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "your-dynamodb-table`

Replace your-s3-bucket-name and your-dynamodb-table with the name of the S3 bucket and DynamoDB table you created, and make sure region matches the AWS region you are working on.

See Terraform S3 state backend documentation for details.

5. Check the config variables

Check the config variables for your Flink application inside terraform/config.tfvars.json and change as desired.

In particular, make sure that the name of the S3 bucket, the Kinesis data stream, and the AWS region match those you are using.

6. Run the deployment container

We can now run the deployment.

docker run --env-file .env.docker --rm -it \
  -v ./flink:/home/flink-project/flink \
  -v ./terraform:/home/flink-project/terraform \
  -v ./build.sh:/home/flink-project/build.sh \
  msf-terraform bash build.sh apply

This command:

• Runs the MSF Terraform container with AWS credentials
• Mounts necessary local directories into the container
• Executes the deployment script build.sh
• Builds the JAR file of the Flink application, uploads it to Amazon S3 and creates the required AWS resources using Terraform

Note that you have to pass the desired Terraform command at the end of the Docker run command, e.g. init, plan, apply or destroy.

7. Update the deployment container

Update a config variable inside terraform/config.tfvars.json and simply run:

docker run --env-file .env.docker --rm -it \
  -v ./flink:/home/flink-project/flink \
  -v ./terraform:/home/flink-project/terraform \
  -v ./build.sh:/home/flink-project/build.sh \
  msf-terraform bash build.sh apply

8. Destroy the deployment container and resources

Run the following command to destroy the created resources:

docker run --env-file .env.docker --rm -it \
  -v ./flink:/home/flink-project/flink \
  -v ./terraform:/home/flink-project/terraform \
  -v ./build.sh:/home/flink-project/build.sh \
  msf-terraform bash build.sh destroy

Note that you have to destroy your DynamoDB table as well as the S3 separately.

Run the following command to delete the created Docker image:

docker image rm msf-terraform

Known simplifications and limitations

This example is for demonstrations purposes and allows you to simulate different phases of the application lifecycle. For simplicity, both application code, configuration, and Terraform code are in the same repository. Also, we have a single build.sh script that builds the Apache Flink application, uploads a new JAR to S3, and executes Terraform, regardless of the changes.

In a real-world CI/CD you probably separate building the application and uploading the JAR file, and trigger it only on specific actions (e.g. a Git commit or merge to the application code).

Also, this example uses the build timestamp appended to the JAR file name to version it. In a real-world application you should use a JAR naming related to the Git commit, or other versioning, that links a unique JAR to a unique state of the code, following the best practices you use for your software development lifecycle (SDLC). We do not recommend to overwrite the same JAR file, always using the same name.

The example also deliberately hardwires the AWS region to keep the example simple. The region can be parametrized using some additional parameter and templating mechanism.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

Authors

• Felix John
• Lorenzo Nicora
• Hamza Khalid