diff --git a/content/karpenter/040_karpenter/install_karpenter.md b/content/karpenter/040_karpenter/install_karpenter.md index 57fc409e..471a6446 100644 --- a/content/karpenter/040_karpenter/install_karpenter.md +++ b/content/karpenter/040_karpenter/install_karpenter.md @@ -57,5 +57,3 @@ kubectl get deployment -n karpenter {{% notice note %}} You can increase the number of Karpenter replicas in the deployment for resilience. Karpenter will elect a leader controller that in charge of running operations. {{% /notice %}} - - diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index 3c8c487d..5f8f7897 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -96,3 +96,11 @@ The Core Instance Fleet launched one xlarge instance, able to run one executor. {{%expand "Question: Did you see more than 9 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} Your Spark application was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. {{% /expand%}} + +#### Managed Scaling in Action + +1. In your EMR cluster page, in the AWS Management Console, go to the **Steps** tab. +1. Go to the **Events** tab to see the scaling events. +![scalingEvent](/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png) + +EMR Managed Scaling constantly monitors [key metrics](https://docs.aws.amazon.com/emr/latest/ManagementGuide/managed-scaling-metrics.html) and automatically increases or decreases the number of instances or units in your cluster based on workload. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 627a8383..dcf62bd1 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -32,6 +32,26 @@ Since our executor size is 4 vCPUs, and each instance counts as the number of it ![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png) -click **Next** to continue to the next steps of launching your EMR cluster. +### Enabling cluster scaling + +While you can always manually adjust the number of core or task nodes (EC2 instances) in your Amazon EMR cluster, you can also use the power of EMR auto-scaling to automatically adjust the cluster size in response to changing workloads without any manual intervention. + +Let's enable scaling for this cluster using **[Amazon EMR Managed Scaling](https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/)**. With EMR Managed scaling you specify the minimum and maximum compute limits for your cluster and Amazon EMR automatically resizes EMR clusters for best performance and resource utilization. EMR Managed Scaling constantly monitors key metrics based on workload and optimizes the cluster size for best resource utilization + +{{% notice note %}} +EMR Managed Scaling is supported for Apache Spark, Apache Hive and YARN-based workloads on Amazon EMR versions 5.30.1 and above. +{{% /notice %}} +1. Select the checkbox for **Enable Cluster Scaling** in **Cluster scaling** section. +1. Set **MinimumCapacityUnits** to **36**, which includes core node capacity units plus capacity units for 8 task executors. +1. Set **MaximumCapacityUnits** to **68**, keeping same capacity units for core nodes but allowing scaling for task nodes. +1. Set **MaximumOnDemandCapacityUnits** to **0**, use EC2 Spot instances only for both Code Nodes and Task Nodes. +1. Set **MaximumCoreCapacityUnits** to **4**, keeping same capacity units for core nodes. +![emrmanagedscaling](/images/running-emr-spark-apps-on-spot/emrmanagedscaling.png) +{{% notice note %}} +Managed Scaling now also has the capability to prevent scaling down instances that store intermediate shuffle data for Apache Spark. Intelligently scaling down clusters without removing the instances that store intermediate shuffle data prevents job re-attempts and re-computations, which leads to better performance, and lower cost. +**[Click here](https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-emr-managed-scaling-shuffle-data-aware/)** for more details. +{{% /notice %}} + +click **Next** to continue to the next steps of launching your EMR cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/scaling_emr_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/scaling_emr_cluster.md deleted file mode 100644 index a1141ed6..00000000 --- a/content/running_spark_apps_with_emr_on_spot_instances/scaling_emr_cluster.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: "Scaling EMR cluster" -weight: 95 ---- - -While you can always manually adjust the number of core or task nodes (EC2 instances) in your Amazon EMR cluster, you can also use the power of EMR auto-scaling to automatically adjust the cluster size in response to changing workloads without any manual intervention. - -In this section, we are going to enable automatic scaling for the cluster using **[Amazon EMR Managed Scaling](https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/)**. With EMR Managed scaling you specify the minimum and maximum compute limits for your cluster and Amazon EMR automatically resizes EMR clusters for best performance and resource utilization. EMR Managed Scaling constantly monitors key metrics based on workload and optimizes the cluster size for best resource utilization. - -{{% notice note %}} -EMR Managed Scaling is supported for Apache Spark, Apache Hive and YARN-based workloads on Amazon EMR versions 5.30.1 and above. -{{% /notice %}} - -### Enable Managed Scaling - -1. In your EMR cluster page, in the AWS Management Console, go to the **Summary** tab. -1. Copy the **ID** from the **Summary** tab. -1. Open the shell terminal in your Cloud9 enivronment that you created at the beginning of this workshop. -1. Run the command after replacing the **CLUSTER-ID** with the one you copied earlier. - -```Bash -aws emr put-managed-scaling-policy \ - --cluster-id CLUSTER-ID \ - --managed-scaling-policy "ComputeLimits={ - UnitType=InstanceFleetUnits, - MinimumCapacityUnits=8, - MaximumCapacityUnits=16, - MaximumOnDemandCapacityUnits=0, - MaximumCoreCapacityUnits=8 - }" -``` - -In the command above, we have set the : - -* Mimumum and maximum capacity for the worker nodes to 8 and 16 respectively. -* **MaximumOnDemandCapacityUnits** to 0 to only use EC2 Spot instances. -* **MaximumCoreCapacityUnits** to 8 to allow scaling of core nodes. - - -### Managed Scaling in Action - -Now we want to post more jobs to the cluster and to trigger scaling. - -1. In your EMR cluster page, in the AWS Management Console, go to the **Steps** tab. -1. Select the Spark application that you created during cluster creation and click **Clone step** -![jobCloning](/images/running-emr-spark-apps-on-spot/emrsparkjobcloning.png) -1. On the next screen, click **Add**. Wait for couple of moments so that step is in **Running** state. -1. Go to the **Events** tab to see the scaling events. -![scalingEvent](/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png) - -With this configuration, Your EMR cluster with automatically scale based on compute requirements of the job and add EC2 spot instances while keeping the limits into consideration. - -{{% notice note %}} -Managed Scaling now also has the capability to prevent scaling down instances that store intermediate shuffle data for Apache Spark. Intelligently scaling down clusters without removing the instances that store intermediate shuffle data prevents job re-attempts and re-computations, which leads to better performance, and lower cost. -**[Click here](https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-emr-managed-scaling-shuffle-data-aware/)** for more details. -{{% /notice %}} \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 06d6d199..9c26b799 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -31,7 +31,7 @@ For the purpose of this workshop we will select instances based on below criteri * Instances which have vCPU to Memory ratio of 1:8, same as R Instance family * Instances with CPU Architecture x86_64 and no GPU Instances * Instances that belong to current generation - * Instances types that are not supported by EMR such as R5N, R5ad and R5b. Enhanced z, I and D Instance families, which are priced higher than R family. So basically, adding a deny list with the regular expression `.*n.*|.*ad.*|.*b.*|^[zid].*`. + * Instances types that are not supported by EMR such as R5N, R5ad and R5b. Enhanced z, I and D Instance families, which are priced higher than R family. So basically, adding a deny list with the regular expression `.*n.*|.*ad.*|.*b.*|^[zid].*`. You can find the list of supported instance types [here](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-supported-instance-types.html). {{% notice info %}} **[Click here](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-supported-instance-types.html)** to find out the instance types that Amazon EMR supports . @@ -58,13 +58,10 @@ r5a.4xlarge r5a.xlarge r5d.2xlarge r5d.4xlarge -r5d.xlarge -r6i.2xlarge -r6i.4xlarge -r6i.xlarge +r5d.xlarge ``` {{% notice note %}} You are encouraged to test what are the options that `ec2-instance-selector` provides and run a few commands with it to familiarize yourself with the tool. For example, try running the same commands as you did before with the extra parameter **`--output table-wide`**. -{{% /notice %}} \ No newline at end of file +{{% /notice %}} diff --git a/static/images/running-emr-spark-apps-on-spot/emrmanagedscaling.png b/static/images/running-emr-spark-apps-on-spot/emrmanagedscaling.png new file mode 100644 index 00000000..45892c89 Binary files /dev/null and b/static/images/running-emr-spark-apps-on-spot/emrmanagedscaling.png differ diff --git a/static/images/running-emr-spark-apps-on-spot/emrsparkjobcloning.png b/static/images/running-emr-spark-apps-on-spot/emrsparkjobcloning.png deleted file mode 100644 index 8609970b..00000000 Binary files a/static/images/running-emr-spark-apps-on-spot/emrsparkjobcloning.png and /dev/null differ diff --git a/static/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png b/static/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png index be0211d2..60ee522e 100644 Binary files a/static/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png and b/static/images/running-emr-spark-apps-on-spot/emrsparkscalingevent.png differ