Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EMR on EKS [WIP] #146

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ weight: 10
# Getting Started
To start the workshop, follow one of the following depending on whether you are...

* ...[running the workshop on your own (in your own account)]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/self_paced.md" >}}), or
* ...[attending an AWS hosted event (using AWS provided hashes)]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/aws_event.md" >}})
* ...[running the workshop on your own (in your own account)]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/self_paced.md" >}}), or
* ...[attending an AWS hosted event (using AWS provided hashes)]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/aws_event.md" >}})

Once you have completed with either setup, continue with **[Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/workspace.md" >}})**
Once you have completed with either setup, continue with **[Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/workspace.md" >}})**
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ weight: 20
{{% notice warning %}}
Only complete this section if you are at an AWS hosted event (such as re:Invent,
Kubecon, Immersion Day, or any other event hosted by an AWS employee). If you
are running the workshop on your own, go to: [Start the workshop on your own]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/self_paced.md" >}}).
are running the workshop on your own, go to: [Start the workshop on your own]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/self_paced.md" >}}).
{{% /notice %}}

### Login to the AWS Workshop Portal
Expand Down Expand Up @@ -42,7 +42,7 @@ We have deployed the below resources required to get started with the workshop u

+ An EKS cluster with the name `eksworkshop-eksctl` and a [EKS managed node group](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) with 2 on-demand instances.

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/eksctl/create_eks_cluster_eksctl_command.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/020_eksctl/create_eks_cluster_eksctl_command.md" %}}


#### Use your resources
Expand All @@ -53,7 +53,7 @@ In this workshop, you'll need to reference the resources created by the CloudFor

1. In the stack details pane, click the **Outputs** tab.

![cnf_output](/images/using_ec2_spot_instances_with_eks/prerequisites/cnf_output.png)
![cnf_output](/images/using_ec2_spot_instances_with_eks/010_prerequisites/cnf_output.png)

It is recommended that you keep this window open so you can easily refer to the outputs and resources throughout the workshop.

Expand All @@ -65,9 +65,9 @@ you will notice additional Cloudformation stacks were also deployed which is the

- Click on the url against `Cloud9IDE` from the outputs

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/workspace_at_launch.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/workspace_at_launch.md" %}}

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/update_workspace_settings.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/update_workspace_settings.md" %}}

### Validate the IAM role {#validate_iam}

Expand All @@ -78,10 +78,10 @@ aws sts get-caller-identity

```

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/at_an_aws_validaterole.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/at_an_aws_validaterole.md" %}}

{{% notice note %}}
Since we have already setup the prerequisites, **you can head straight to [Test the Cluster]({{< relref "/using_ec2_spot_instances_with_eks/eksctl/test.md" >}})**
Since we have already setup the prerequisites, **you can head straight to [Test the Cluster]({{< relref "/using_ec2_spot_instances_with_eks/020_eksctl/test.md" >}})**
{{% /notice %}}

{{% /expand%}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ weight: 10
---

{{% notice warning %}}
Only complete this section if you are running the workshop on your own. If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, etc), go to [Start the workshop at an AWS event]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/aws_event.md" >}}).
Only complete this section if you are running the workshop on your own. If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, etc), go to [Start the workshop at an AWS event]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/aws_event.md" >}}).
{{% /notice %}}

### Running the workshop on your own
Expand Down Expand Up @@ -34,4 +34,4 @@ as an IAM user with administrator access to the AWS account:
![Login URL](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-4-save-url.png)


Once you have completed the step above, **you can head straight to [Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/workspace.md" >}})**
Once you have completed the step above, **you can head straight to [Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/workspace.md" >}})**
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ chapter: false
weight: 60
---

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/update_workspace_settings.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/update_workspace_settings.md" %}}

We should configure our aws cli with our current region as default:
```
Expand All @@ -17,5 +17,5 @@ aws configure set default.region ${AWS_REGION}
aws configure get default.region
```

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/validate_workspace_role.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/validate_workspace_role.md" %}}

Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,17 @@ weight: 20


{{% notice warning %}}
**DO NOT PROCEED** with this step unless you have [validated the IAM role]({{< relref "../prerequisites/update_workspaceiam.md#validate_iam" >}}) in use by the Cloud9 IDE. You will not be able to run the necessary kubectl commands in the later modules unless the EKS cluster is built using the IAM role.
**DO NOT PROCEED** with this step unless you have [validated the IAM role]({{< relref "../010_prerequisites/update_workspaceiam.md#validate_iam" >}}) in use by the Cloud9 IDE. You will not be able to run the necessary kubectl commands in the later modules unless the EKS cluster is built using the IAM role.
{{% /notice %}}

#### Challenge:
**How do I check the IAM role on the workspace?**

{{%expand "Expand here to see the solution" %}}

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/validate_workspace_role.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/validate_workspace_role.md" %}}

If you do not see the correct role, please go back and **[validate the IAM role]({{< relref "../prerequisites/update_workspaceiam.md" >}})** for troubleshooting.
If you do not see the correct role, please go back and **[validate the IAM role]({{< relref "../010_prerequisites/update_workspaceiam.md" >}})** for troubleshooting.

If you do see the correct role, proceed to next step to create an EKS cluster.
{{% /expand %}}
Expand All @@ -26,7 +26,7 @@ If you do see the correct role, proceed to next step to create an EKS cluster.

The following command will create an eks cluster with the name `eksworkshop-eksctl`. It will also create a nodegroup with 2 on-demand instances.

{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/eksctl/create_eks_cluster_eksctl_command.md" %}}
{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/020_eksctl/create_eks_cluster_eksctl_command.md" %}}

eksctl allows us to pass parameters to initialize the cluster. While initializing the cluster, eksctl does also allow us to create nodegroups.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ draft: false
---
### Selecting instances for Spot Instance Diversification

[See "Selecting Instance Types" under Spot managed node groups]({{< relref "/using_ec2_spot_instances_with_eks/spotmanagednodegroups/selecting_instance_types.md" >}})
[See "Selecting Instance Types" under Spot managed node groups]({{< relref "/using_ec2_spot_instances_with_eks/040_spotmanagednodegroups/selecting_instance_types.md" >}})

Once you finish selecting instance types, proceed to "Adding Spot Workers with eksctl" section to continue creating Self Managed Spot node groups.
130 changes: 130 additions & 0 deletions content/using_ec2_spot_instances_with_eks/090_emroneks/010_prereqs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: "Prerequisites"
date: 2021-03-15T16:24:50-04:00
weight: 10
draft: false
---


In this chapter, we will prepare your EKS cluster so that it is integrated with EMR on EKS.

{{% notice note %}}
If you don't have EKS cluster with Spot Managed Node Groups then please review instructions from **[start the workshop]({{< relref "../010_prerequisites" >}})**, **[launch using eksctl]({{< relref "../020_eksctl" >}})**, **[create Spot managed node groups]({{< relref "../040_spotmanagednodegroups" >}})** and **[deploy cluster autoscaler]({{< relref "../070_scaling/deploy_ca.md" >}})** modules.
{{% /notice %}}

### Create namespace and RBAC permissions

Let's create a namespace '**spark**' in our EKS cluster. After this, we will use the automation powered by eksctl for creating RBAC permissions and for adding EMR on EKS service-linked role into aws-auth configmap

```
kubectl create namespace spark

eksctl create iamidentitymapping --cluster eksworkshop-eksctl --namespace spark --service-name "emr-containers"
```

### Enable IAM Roles for Service Account (IRSA)

Your cluster should already have OpenID Connect provider URL. Only configuration that is needed is to associate IAM with OIDC. You can do that by running this command

```
eksctl utils associate-iam-oidc-provider --cluster eksworkshop-eksctl --approve
```

### Create IAM Role for job execution

Let's create the role that EMR will use for job execution. This is the role, EMR jobs will assume when they run on EKS.

```
cat <<EoF > ~/environment/emr-trust-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "elasticmapreduce.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EoF

aws iam create-role --role-name EMRContainers-JobExecutionRole --assume-role-policy-document file://~/environment/emr-trust-policy.json

```

Next, we need to attach the required IAM policies to the role so it can write logs to s3 and cloudwatch.
```
cat <<EoF > ~/environment/EMRContainers-JobExecutionRole.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
EoF
aws iam put-role-policy --role-name EMRContainers-JobExecutionRole --policy-name EMR-Containers-Job-Execution --policy-document file://~/environment/EMRContainers-JobExecutionRole.json
```

### Update trust relationship for job execution role

Now we need to update the trust relationship between IAM role we just created with EMR service identity.

```
aws emr-containers update-role-trust-policy --cluster-name eksworkshop-eksctl --namespace spark --role-name EMRContainers-JobExecutionRole
```

### Register EKS cluster with EMR

The final step is to register EKS cluster with EMR.

```
aws emr-containers create-virtual-cluster \
--name eksworkshop-eksctl \
--container-provider '{
"id": "eksworkshop-eksctl",
"type": "EKS",
"info": {
"eksInfo": {
"namespace": "spark"
}
}
}'
```
After you register, you should get confirmation that your EMR virtual cluster is created. A virtual cluster is an EMR concept which means that EMR service is registered to Kubernetes namespace and it can run jobs in that namespace.

```output
"id": "av6h2hk8fsyu12m5ru8zjg8ht",
"name": "eksworkshop-eksctl",
"arn": "arn:aws:emr-containers:xx-xxxx-x:xxxxxxxxxxxx:/virtualclusters/av6h2hk8fsyu12m5ru8zjg8ht"
```


### Create S3 code bucket
Let's create a s3 bucket to upload sample scripts and logs.

```sh
export s3DemoBucket=s3://emr-eks-demo-${ACCOUNT_ID}-${AWS_REGION}
aws s3 mb $s3DemoBucket
```
116 changes: 116 additions & 0 deletions content/using_ec2_spot_instances_with_eks/090_emroneks/020_deploy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
title: "Run Sample Workload"
date: 2021-05-11T13:38:18+08:00
weight: 20
draft: false
---

### Spark Pod Template
Beginning with Amazon EMR versions `5.33.0 or 6.3.0`, Amazon EMR on EKS supports Spark’s pod template feature. Pod templates are specifications that determine how to run each pod. You can use pod template files to define the driver or executor pod’s configurations that Spark configurations do not support.

{{% notice info %}}
For more information about the pod templates support in EMR on EKS, see [Pod Templates](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/pod-templates.html).
{{% /notice %}}

To reduce costs, you can schedule Spark driver tasks to run on On-Demand instances while scheduling Spark executor tasks to run on Spot instances.

With pod templates you can define label `eks.amazonaws.com/capacityType` as a [node selector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/), so that you can schedule Spark driver pods on On-demand Instances and Spark executor pods on the Spot Instances.

Now, you will create a sample pod template for Spark Driver. Using nodeSelector `eks.amazonaws.com/capacityType: ON_DEMAND` this will run on On-demand Instances.
```sh
cat > spark_driver_pod_template.yml <<EOF
apiVersion: v1
kind: Pod
spec:
volumes:
- name: source-data-volume
emptyDir: {}
- name: metrics-files-volume
emptyDir: {}
nodeSelector:
eks.amazonaws.com/capacityType: ON_DEMAND
containers:
- name: spark-kubernetes-driver # This will be interpreted as Spark driver container
EOF
```

Next, you will create a sample pod template for Spark executors. Using nodeSelector `eks.amazonaws.com/capacityType: SPOT` this will run on Spot Instances.
```sh
cat > spark_executor_pod_template.yml <<EOF
apiVersion: v1
kind: Pod
spec:
volumes:
- name: source-data-volume
emptyDir: {}
- name: metrics-files-volume
emptyDir: {}
nodeSelector:
eks.amazonaws.com/capacityType: SPOT
containers:
- name: spark-kubernetes-executor # This will be interpreted as Spark executor container
EOF
```

Let's upload sample pod templates and python script to s3 bucket.

```sh
aws s3 cp threadsleep.py ${s3DemoBucket}
aws s3 cp spark_driver_pod_template.yml ${s3DemoBucket}/pod_templates/
aws s3 cp spark_executor_pod_template.yml ${s3DemoBucket}/pod_templates/
```

Next we submit the job.

```sh
#Get required virtual cluster-id and role arn
export VIRTUAL_CLUSTER_ID=$(aws emr-containers list-virtual-clusters --query "virtualClusters[?state=='RUNNING'].id" --output text)

export EMR_ROLE_ARN=$(aws iam get-role --role-name EMRContainers-JobExecutionRole --query Role.Arn --output text)

#start spark job with start-job-run
aws emr-containers start-job-run \
--virtual-cluster-id $VIRTUAL_CLUSTER_ID \
--name pi-spot \
--execution-role-arn $EMR_ROLE_ARN \
--release-label emr-5.33.0-latest \
--job-driver '{
"sparkSubmitJobDriver": {
"entryPoint": "'${s3DemoBucket}'/threadsleep.py",
"sparkSubmitParameters": "--conf spark.kubernetes.driver.podTemplateFile=\"'${s3DemoBucket}'/pod_templates/spark_driver_pod_template.yml\" --conf spark.kubernetes.executor.podTemplateFile=\"'${s3DemoBucket}'/pod_templates/spark_executor_pod_template.yml\" --conf spark.executor.instances=15 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"}}' \
--configuration-overrides '{
"applicationConfiguration": [
{
"classification": "spark-defaults",
"properties": {
"spark.dynamicAllocation.enabled": "false",
"spark.kubernetes.executor.deleteOnTermination": "true"
}
}
],
"monitoringConfiguration": {
"cloudWatchMonitoringConfiguration": {
"logGroupName": "/emr-on-eks/eksworkshop-eksctl",
"logStreamNamePrefix": "pi"
},
"s3MonitoringConfiguration": {
"logUri": "'${s3DemoBucket}'/"
}
}
}'
```

You will be able to see the completed job in EMR console.

Let's check the pods deployed on On-Demand Instances and should now see Spark driver pods running on On-Demand instances.

```sh
for n in $(kubectl get nodes -l eks.amazonaws.com/capacityType=ON_DEMAND --no-headers | cut -d " " -f1); do echo "Pods on instance ${n}:";kubectl get pods -n spark --no-headers --field-selector spec.nodeName=${n} ; echo ; done
```

Let's check the pods deployed on Spot Instances and should now see Spark executor pods running on Spot Instances.

```sh
for n in $(kubectl get nodes -l eks.amazonaws.com/capacityType=SPOT --no-headers | cut -d " " -f1); do echo "Pods on instance ${n}:";kubectl get pods -n spark --no-headers --field-selector spec.nodeName=${n} ; echo ; done
```

Loading