Merge pull request #179 from bpguasch/master

Integrated AWS Step Functions for workflow orchestration
awslabs · Aug 3, 2022 · 6bdfe98 · 6bdfe98
2 parents 80b1f3c + a6239ee
commit 6bdfe98
Show file tree

Hide file tree

Showing 20 changed files with 765 additions and 597 deletions.
diff --git a/content/rendering-with-batch/_index.md b/content/rendering-with-batch/_index.md
@@ -7,11 +7,11 @@ pre: "<b>9. </b>"
 ---
 
 {{% notice info %}}
-The estimated completion time of this lab is **60 minutes**. Please note that rendering the animation presented below can incur in costs up to **$15**.
+The estimated completion time of this lab is **90 minutes**. Please note that rendering the animation presented below can incur in costs up to **$15**.
 {{% /notice %}}
 ## Overview
 
-In this workshop you will learn to submit jobs with [AWS Batch](https://aws.amazon.com/batch/) following Spot best practices to [render](https://en.wikipedia.org/wiki/Rendering_(computer_graphics)) a [Blender](https://www.blender.org/) file in a distributed way. You will be creating a docker container and publishing it in Amazon Elastic Container Registry (ECR). Then you will use that container in AWS Batch using a mix of EC2 On-Demand and Spot instances. [Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) are EC2 spare capacity offered at steep discounts compared to On-Demand instances and are a cost-effective choice for applications that can be interrupted, what makes them well-suited for the batch processing that we will run. After going through all the sections, you will have the following pipeline created:
+In this workshop you will learn to submit jobs with [AWS Batch](https://aws.amazon.com/batch/) following Spot best practices to [render](https://en.wikipedia.org/wiki/Rendering_(computer_graphics)) a [Blender](https://www.blender.org/) file in a distributed way. You will be creating a docker container and publishing it in Amazon Elastic Container Registry (ECR). Then you will use that container in AWS Batch using a mix of EC2 On-Demand and Spot instances. [Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) are EC2 spare capacity offered at steep discounts compared to On-Demand instances and are a cost-effective choice for applications that can be interrupted, what makes them well-suited for the batch processing that we will run. After going through all the sections, you will have the following pipeline created, orchestrated by AWS Step Functions:
 
 1. A python script downloads the Blender file from S3 to extract the number of frames from the Blender project.
 2. The script submits a batch job using an `array job` with as many tasks as number of frames. It also submits a single stitching job using [FFmpeg](https://ffmpeg.org/) to create a final video file.

diff --git a/content/rendering-with-batch/batch/batch.files/job_submission.py b/content/rendering-with-batch/batch/batch.files/job_submission.py
diff --git a/content/rendering-with-batch/batch/batch.files/pottery.blend b/content/rendering-with-batch/batch/batch.files/pottery.blend
diff --git a/content/rendering-with-batch/batch/job_definition.md b/content/rendering-with-batch/batch/job_definition.md
@@ -18,7 +18,8 @@ cat <<EoF > job-definition-config.json
     "containerProperties": {
         "image": "${IMAGE}",
         "vcpus": 1,
-        "memory": 8000
+        "memory": 8000,
+        "command": ["Ref::action", "-i", "Ref::inputUri", "-o", "Ref::outputUri", "-f", "Ref::framesPerJob"]
     },
     "retryStrategy": {
         "attempts": 3
@@ -36,6 +37,7 @@ Let's explore the configuration parameters in the structure:
 - **image**: the image used to start a container, this value is passed directly to the Docker daemon.
 - **vcpus**: The number of vCPUs reserved for the job. Each vCPU is equivalent to 1,024 CPU shares.
 - **memory**: hard limit (in MiB) for a container. If your container attempts to exceed the specified number, it's terminated.
+- **command**: this is the command that will be executed in the container when the job is started. It has placeholders for some parameters that will be substituted when submitting the job using AWS Batch.
 - **platformCapabilities**: the platform capabilities required by the job definition. Either `EC2` or `FARGATE`.
 
 {{% notice info %}}

diff --git a/content/rendering-with-batch/batch/job_queue.md b/content/rendering-with-batch/batch/job_queue.md
@@ -42,7 +42,8 @@ All the compute environments within a queue must be either (`SPOT` and/or `EC2`)
 Execute this command to create the job queue. To learn more about this API, see [create-job-queue CLI command reference](https://docs.aws.amazon.com/cli/latest/reference/batch/create-job-queue.html).
 
 ```
-aws batch create-job-queue --cli-input-json file://job-queue-config.json
+export JOB_QUEUE_ARN=$(aws batch create-job-queue --cli-input-json file://job-queue-config.json | jq -r '.jobQueueArn')
+echo "Job queue Arn: ${JOB_QUEUE_ARN}"
 ```
 
 Next, you are going to create a **Job Definition** that will be used to submit jobs.