From 5449f7da03d0387ca2a8adb241ee275f0dce29d0 Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 17 Jun 2019 15:51:13 +0000 Subject: [PATCH 01/40] committing emr into a new branch --- .../_index.md | 2 +- .../automations_monitoring.md | 14 +++++++------- .../fleet_config_options.md | 4 ++-- .../launching_emr_cluster-1.md | 6 +++--- .../launching_emr_cluster-2.md | 2 +- .../right_sizing_executors.md | 4 ++-- .../selecting_instance_types.md | 6 +++++- .../spot_savings_summary.md | 4 ++-- .../tracking_spot_interruptions.md | 4 ++-- .../verifying_results.md | 4 ++-- .../visualizing_costs.md | 16 +++++++++++++++- .../sparksubmit.png | Bin 14817 -> 0 bytes .../sparksubmitstep.png | Bin 0 -> 15091 bytes 13 files changed, 42 insertions(+), 24 deletions(-) delete mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmit.png create mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 3559fa2c..a353d30c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -29,4 +29,4 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools, as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md index cb855001..2f835496 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md @@ -12,17 +12,17 @@ In this section we will simply look at a CLI command that can be used to start a 1. In the AWS Management Console, under the EMR service, go to your cluster, and click the **AWS CLI export** button. 2. Find the --instance-fleets parameter, and copy the contents of the parameter including the brackets: ![cliexport](/images/running-emr-spark-apps-on-spot/cliexport.png) -3. Paste the data into a JSON validator like [JSON Lint] (https://jsonlint.com/) and vlaidate the JSON file. this will make it easy to see the Instance Fleets configuration we configured in the console, in a JSON format, that can be re-used when you launch your cluster programmatically. +3. Paste the data into a JSON validator like [JSON Lint] (https://jsonlint.com/) and validate the JSON file. this will make it easy to see the Instance Fleets configuration we configured in the console, in a JSON format, that can be re-used when you launch your cluster programmatically. #### (Optional) Set up CloudWatch Events for Cluster and/or Step failures Much like we set up a CloudWatch Event rule for EC2 Spot Interruptions to be sent to our email via an SNS notification, we can also set up rules to send out notifications or perform automations when an EMR cluster fails to start, or a Task on the cluster fails. This is useful for monitoring purposes. -In this example, let's set up a notification for when our EMR step failed. +In this example, let's set up a notification for when our EMR step failed.\ 1. In the AWS Management Console, go to Cloudwatch -> Events -> Rules and click **Create Rule**.\ -2. Under Service Name select EMR, and under Event Type select State Change.\ -3. Check **Specific detail type(s) and from the dropdown menu, select **EMR Step Status Change**\ -4. Check Specific states(s) and from the dropdown menu, select **FAILED**.\ +2. Under Service Name select EMR, and under Event Type select **State Change**.\ +3. Check **Specific detail type(s)** and from the dropdown menu, select **EMR Step Status Change**\ +4. Check **Specific states(s)** and from the dropdown menu, select **FAILED**.\ ![cwemrstep](/images/running-emr-spark-apps-on-spot/emrstatechangecwevent.png) -5. In the targets menu, click **Add target**, select SNS topic and from the dropdown menu, select the SNS topic you created and click **Configure details**.\ -6. Provide a name for the rule and click **Create rule**\ +5. In the targets menu, click **Add target**, select **SNS topic** and from the dropdown menu, select the SNS topic you created and click **Configure details**.\ +6. Provide a name for the rule and click **Create rule**.\ 7. You can test that the rule works by following the same steps to start a cluster, but providing a bad parameter when submitting the step, for example - a non existing location for the Spark application or results bucket. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 3fc0be60..df0384b8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -4,12 +4,12 @@ weight: 85 draft: true --- -While our cluster is starting (7-8 minutes) and the job is running (~5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) #### Maximum Spot price -Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot reuqests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the EC2 Spot console under **Pricing History**. +Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) are 64, then 16 * r4.xlarge instances will be launched by EMR for our Core fleet. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index d8fb0ad3..ff5972e3 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -4,7 +4,7 @@ weight: 60 draft: true --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the job we submit to it, and will run solely on Spot Instances. The job is a simple word count application that will run against a public data set of Amazon product reviews. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. To launch the cluster, follow these steps:\ @@ -13,7 +13,7 @@ To launch the cluster, follow these steps:\ 1. Click "**Go to advanced options**"\ 1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark**.\ 1. Under "**Add steps (Optional)**" -> Step type drop down menu select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ +**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ (make sure you have two '-' chars) **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: *aws s3 cp \ s3://\* ```python @@ -28,7 +28,7 @@ Then add the location of the file under the **Application location** field, i.e: **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ **Action on failure**: Leave this on *Continue* and click **Add** to save the step. -![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmit.png) +![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) Check the **Auto-terminate cluster after the last step is completed** option. Since we are looking to run a transient cluster just for running our Spark application, this will terminate the cluster once our submitted step (Spark Application) has completed. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 3bee6b8c..e1e4908b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -10,7 +10,7 @@ Let's discuss the right setup for each of our node types:\ #### **Master node**: Unless your cluster is very short-lived and the runs are cost-driven, avoid running your Master node on a Spot Instance. We suggest this because a Spot interruption on the Master node terminates the entire cluster. \ For the purpose of this workshop, we will run the Master node on a Spot Instance as we simulate a relatively short lived job running on a transient cluster. There will not be business impact if the job fails due to a Spot interruption and later re-started.\ -Click **Add / remove instance types to fleet** and select two relatively small and cheap instance types - i.e c4.large and m4.large and check Spot under target capacity. EMR will only provision one instance, but will select the best instance type for the Master node. +Click **Add / remove instance types to fleet** and select two relatively small and cheap instance types - i.e c4.large and m4.large and check Spot under target capacity. EMR will only provision one instance, but will select the best instance type for the Master node based on price and available capacity. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleets-master.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 23bc7993..043aec6e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -14,7 +14,7 @@ If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid goi EMR by default places limits on executor sizes in two different ways, this is in order to avoid having the executor consume too much memory and interfere with the operating system and other processes running on the instance. -1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the spark., +1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the default YARN configuration options. Let's have a look at a few examples of instances that have our approximate vCPU:Mem ratio:\ r4.xlarge: yarn.scheduler.maximum-allocation-mb 23424\ r4.2xlarge: yarn.scheduler.maximum-allocation-mb 54272\ @@ -26,4 +26,4 @@ So we can conclude that if we decrease our executor size to ~18GB, we'll be able ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) -Our conclusion is that in order to use R family instances with the flexibility of also using the smallest instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file +Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 7868fa9a..967c8dfd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -11,10 +11,14 @@ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aw We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ -For example: r4.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ +For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ However, at the time of writing, when looking at the EU (Ireland) region in the Spot Instance advisor, the r5.2xlarge instance type is showing an interruption rate of >20%.\ Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. At the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge +{{% notice note %}} +Spot Instance interruption rates are dynamic, the above just provides a real world example from a specific time and would probably be different when you are performing this workshop. +{{% /notice %}} + To keep our flexibility in place and be able to provide multiple instance types for our EMR cluster, we need to make sure that our executor size will be under the EMR YARN limitation that we saw in the previous step, **Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 18 GB of RAM, and also have low Spot interruption rates (maximum 10-15%). diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 24b87c64..da658e88 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -7,7 +7,7 @@ draft: true When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. 1. Open the AWS Management console and go to the EC2 Service in the region where you are running your EMR cluster. -2. On the left pane, click Spot Requests. -3. In the main pane, click Savings Summary +2. On the left pane, click **Spot Requests**. +3. In the main pane, click **Savings Summary**. ![savingssummary](/images/running-emr-spark-apps-on-spot/savingssummary.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index 5c2f70ab..3182f572 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -4,14 +4,14 @@ weight: 100 draft: true --- -So we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. +Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions, but when we get started with EMR jobs this could be useful. {{% /notice %}} -Let's set up an email notification for when Spot interruptions occur, so if something fails in our EMR jobs we'll be able to check if the failures correlate to a Spot interruption. +Let's set up an email notification for when Spot interruptions occur, so if there are any failures in our EMR applications, we'll be able to check if the failures correlate to a Spot interruption. #### Creating an SNS topic for the notifications 1. Create a new SNS topic and take note of the topic ARN that is returned diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index a9c0b36d..7e56363c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -1,10 +1,10 @@ --- -title: "Verifying the job's results" +title: "Verifying the app's results" weight: 140 draft: true --- -In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that the job finished successfully. +In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. 1. In the AWS Management Console, go to the [Athena service] (https://console.aws.amazon.com/athena/home) and verify that you are in the correct region where you ran your EMR cluster. 2. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index 15680794..0f79768d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -4,4 +4,18 @@ weight: 130 draft: true --- -In this section we will use AWS Cost explorer to look at the costs of our EMR runs. Need to think about this step because data is only available after 2 days. \ No newline at end of file +In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. +{{% notice note %}} +It will take 24-48 hours for your usage to appear in Cost Explorer, so you can plan to come back to this step later to check the costs of running the workshop. If your organization administrator has not granted you access to Billing information, then you will not be able to access Cost Explorer, but you can look at the examples provided below. +{{% /notice %}} + +In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=Name, Value=EMRTransientCluster1. These tags can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. + + +### Analyzing costs with AWS Cost Explorer +[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. Get started quickly by creating custom reports (including charts and tabular data) that analyze cost and usage data, both at a high level (e.g., total costs and usage across all accounts) and for highly-specific requests (e.g., m2.2xlarge costs within account Y that are tagged “project: secretProject”). Using AWS Cost Explorer, you can dive deeper into your cost and usage data to identify trends, pinpoint cost drivers, and detect anomalies. + +Let's use Cost Explorer to analyze the costs of running our EMR application.\ +1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ +2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**MyWorkshopEMRTransientCluster1**" +3. To diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmit.png b/static/images/running-emr-spark-apps-on-spot/sparksubmit.png deleted file mode 100644 index 5e31e0358f2f7d08bc459219a625c62e1a889bbf..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 14817 zcmeHucUV(hwegz+H3vRFZbLL%%b^s60Hnt!?_sX-OHyrNAwvdZf<~*fc^0V9%UT@HO5D!mz zn((FvANO1+=*;C19v(5w*8eta;Ja%)JVxQxr$7#o?hB*m)27>D+N8~c^nD}G(6^=a zhNJ9HYOnI$B})Me+zAgaw6_jwX&o_;=D!|si4 zIQqQU|J(CJXngVsktd+jypS&I<%7oQ8zv`jcUT;knKF{)K$aZBAe&4yhdFM>o-`AC z{%X1O+d`uu72wO$ILy65}3e`|wB*)#};mgkdM;FDQ{4PVmcENj!m|NiWX z87Jg#&rY|RvHHz^YLs8D4H#_*BM*F_da)*yI7>?8z32Y4Z_Qfdyw`tqDb86GPqN2@ z(HlQ8>1yD)H5xUInqEysG!y8>W1F+RXJQ}TTKdpwT4z8-Ad^KzJ~rbCJijAd>v7bd06>A+}=?_B3 z^Sce#K*HOOnAUQ|U*IE-y0SSTjW*n5jd5VOHXJ%)-9#$lzQqax9!;8-mshzXm8 zF+gOO(xK~?#-)adR8~L;YLKW6-(B`$7P`s!;QD7V#ekbfIxcz5PrKr&XK`gRj351uR65vPqcv>cQ?L#1Fbc+BwL+8mc0& zWKD?qbHM-|yzaqbh>MS{cP_f@b%X0xK!%5*&(lC#DqEE_aFo3k$f;p_+}gmr=)_ss zFBm`uwoL@0`|oC|fq^EmwLrFjzo1W?ElHdI!7c#Yjy@*9Re+Fd^0eioq!<1lqc$-0 zG;why@ajsih(DSc%w%8}WHLYM!vC?RNp(%@8{G)^G8-+<6dfP5Y^)@WXWrgr-qwv^ ztNFLf)X*WDd8S7dPxvXpy{9&>L>_V3TgrYzpbBgl6)*NZti4grNke$oeF$(ws>FH` zYpZ^@*q(0yl1(kJ>297~KwADdccIhQs&(5Pwvp)gveJeHM<$4ImxQ^AT=RYXm`&Cf zG*(gs<_^0jlrOr-hWD*tbr9>!4Z{kc-j;>nEgRRf&>LA8=uN~A{+uxuauWaN2DovZ zjAoM=N!W_bE)Ib|heNJnY5i$Rl;w#s_FUQAvyUnM-DFOsC;+{J$gM$dSuRn_|)GjJFJLrg|Md~ihns2_*y0Mr=o`4e0$Dd$fVpT!I5xZ7sE~f7-%CH+#NY?6*Go z#P~@g7=;KI&egv>*>vkRo~wnt3OrY9|FuIGX9O#Ty4YJ$bsHuny2y000qyx-bF=d9v*BX56Z0@H zn~X&jZ9^SCj%KVDCz2IqGq?AqxHD%IL15LsabM;Be29sx0Id z!G!db{_3)lF>#ePgZ%;XA_^|U`sh{c5|c8*3dN&9kE%*4u0Nrrml_D^A0U#Low!MF zuT7WRrn^H8vz!H)&!tnH-5MtPrTQw`!&)nZRbXaUGSpGqAMg|J!K#?x0HleP>!);{ zUA1(+bq9ZlwFdJY)g#>BYghYEXvaoeH1;$I1$9C zOT=Zxwmk`%Vpcy15v@&2Q5xg}B~=y)pK(0a_8@9RDtT=Xb^cs3v^sY$u0UDzfpf8m z74584ZsxH|QrMRvUltDdABoLANUqgh^@#OCV{Z4RYS)x35?cX{*vm@&U+IU|49seR zC{w6`Crbxa4#Jv&xGyn7S&f9{X1{?XddbB=>Cj*0yF=>^=v-apD-==j0-bk-$QRQd z$LTuCe+%JVswNv8z$zMBjQ|I1MZ{K?)%3BLQz4Xj#c-&S6nSRYB3LOvgGRLyao^Cv z3*=l!Z;ZP{*siyEUusT zQ%hdDe;xJ8I(hlyNj1>n)e9*HvwZkdo$b;a0Un26mF89GdZR9>WAjE!Dy^WW4QKKBrJ<@+-u}Ed{zb~7-51cq zW>?M%LxWn3=C+#+y03;f7$;V&U*D#jt1)XVy4~yWJ6u07sTH72T97$y;VI|Bp!Fc& zN~&!GxyH>XO_}q2IiI7_Cv6)D*G&Fs37QXS7!@^b#&4(Yw<1}>cI`GXt zMYt1WjICS;hfx);!6sI+;m@VMR3PW0HO@aq+m@=RPd%Go57G)-QWD)dYQMw4? zum66P=c8+!*#G*$@7L=zU?0~-Qr>m$4!ybq3*WuX`E<{nn3Fx*>h{HnjqHdM`!u>M zbnt#{5?Ku_1}+L`6~NxFQG!-Lk*wvQl_+MP=|K|Dh$tv2hZNtHu7>;Rvbrn2s}Yk- zKPo0I4Qph)84V5Ztj|To;z|+Qfb~VCOLL12n;SOnX2q28%8)fdlY?<$#*g`PicHfk zcpoG+)l=xju6t4@YKR64!5wQ;b>Ez#jzuy}C|UCeVV1V%zFQS7n~b0^Zx8Pz5V2A^ zsZbg$t_}3P#Fgp@uR{1MLBN%d>Tz8}gQRbM=$h86x`H5o=C2CFQgPd;b|1We!Oxdq z9A|%OQf7oUl5n9M_|HA>lrHh z3wt6uhH!AODJJZtpIPUoM2@-Ul(xmm8>r#0@n`W?jatIHRUDck!s?1@laAT(<^V!% z%uI`rsi7=Y56^I$C3)?+T~2`rs$~TFI+FB*m6{#6ESHs+L4g2!-&;UOysybSb?CwK zx9u_%=880uYmG&rt;cR{%q^VixzDWqAOaY5g7}~pG!`VX0qr)XDn%|Ew1Mn-X&nv9 zfmZ#1G|Pr7Y2ybGj1bNCgZOWcF6_joUwy=PZ%C)rZcqH#M#Z%A-B}OwBO9LiWi_`k zMqr1(w!Ng8>8CEcza8Hhd5cMH zc(blDsQJ1cD%u2}3?GY@m~oR?GFDQ>BiyDH1p=hn!*dw2HNbq-Un|C=}LVsYVR{fV;|UdV51%gWZhKPi+I@T{PokbhtP?U>c_s@6&1M> zli`12H#p(s?A0S%)e0(vbmPB_31#-M_9uE8UmEG(Kf<3{U#JGt(H{8bnHyksaqwHA z$`C^cdNwUbM8@9CJJoIST92$dW`2WLVat){L>li2J)%<=`uTvU*?DK5xU-EgP>;~p zkLdx%mYn24w%2f#%nMMy=-%mQiQ^Aot`FZEQwL?REvJ_v zLg#=Mi-xxzj>0={N(EG~z1wz6{+7-T`bl=OOgw0eQ7HYah6iFQ4Do*G7buJvE!q#f zLW-YpV=c9gMOaslzopdfG%4%1(={$wlkYCAm~1@p1}Z1IZ$ZbAY~DShr%KnJhX7lB zQ2c&rIdWFYq)$f$0YT#OOv~>h*zG391;=f+`C0vE1OKI1=h(CD*R_8)`Cp2H=zH9F zX?V2C=zq_(^n=YzW67>#TmGeMq-7Jp9ar=& z8xvCpU!GdL_T~!OC&DN_qU%?-hJxu5Gc zTXWf;y0LM-5prWerjabm|6w!pG{Y9V{<(8};Y`m1uhGi^kej;e8B;^=YOq+17 zC5`J;_CpwbFN%z`VP9bv(E64=QLCzPwuhvs&OIxhgqos7gXwTahnle5S=_09dEl)t~gT*SE+;$+Mu0T*bVVxdiz#qfuN89{#Cr-J>3Vfwf9(mL7GU zTL`{IatEX3$Qaw2K)ITh4MHQp&Cx?+ROzbl7<;XjS^wUDyqfD2UmGZdF6Z@9UPvx( zYGeaRw6|eRkPFc%q)s6mwO^`iXOMH>O*@5nbF2aZq-&P&f;@xIo+Zfoy!1fwHBh11 z4C6c=A@ycwtq5ScUXCaXU>T&ok|=n@4;akb_pT zp{mHlYx()&W|a_EzFJX0NAyrbo?gh66_~uNug9`E&ma_Lx*G_VGWL z2|bOZe4$PFR@@o`oNNsP2}GXc3V<4^0)A!*lM(Eqb$^S&6dsX~;>}S;7d*cG2;yM9I+G0;hV$Z4&b* z+)nT4>*D$e-m9{pq!abUc@pS`hEXSw_`|#%K?d9?X-X{4cJs(h;+D_^eg$l~6+r+` zgay}`{NkGW2ZSs)W2{=KIfm}q9m;cibkF~j!_^&1u+_(0H#O0AD^ASs(j#X)o)JNtJn^5Jz zF@*qNoykZnp_K5<0GFiJ&8!SSn1P|Mwm(0PDB-#h^9unxuMsG~`S}B+rxUEE$vg>j z+q&lmNh40GN3N=`U;%NqiyNKe$0fgu2Ndm5SpBTV|7gpP(Y`qubfhC+J;Ju&s}I_Q z66=6&wJ-=O+KD%Z-Q!TN7h7moYL z$f=bbBBT#ie7bclTY?^~X}AUz`w-dvgw9qS(pCzQKj&8~9`x5dK4@RFw~X3#v|)h% zUd0Wpl<>;c7<ECPhiMCE=|b7sC9{-eTY@s9MkXijO~46AJ=T;&xRVeb0N3S%*?4=UUj zAtJg^-x$-&UkaS6hK1rTuF2vijS7*mp*T|+Z+YBOa5&leSPE+ZE7EpZohj3#J#F}v zS(}o#Qr8@_u^O~QNX9e*K0XBzY2!y?v_{Bva=ueE!^G=y*NzQ(^z1D4spcf3H4sLm*J0TiUi+v=wddRKy_q~bt%?R2G5 zC1Joc`@=ju^vZT~vru74#3PN_JS%FW7Oi^$1y9f5QTC#zaABO4a*RwGT z5-l6WEcYLD6(e|%_AoHLJPC(H#rBRQ-Q?gF;H>5h*}xb=q)oIN3>7+7gZylodqOUB zcej1 znF>z6B7{ButQXh#nY}W^0EQn8b!AXlvbzUrC#4EQft>W!LS-{YM{)R7Qq7KtpHM!QO}t69YxMH z#tkOr_ZA8)txLq)x@1%a%hU9Qc}ESQO6A>0k9}cT#n~9wG>QXQMvPrg#=7m>jKa|s z#UzZc0nso1aQw@ID1sk=#4R{u`f@SaJAtU+V z{dUO*ffww|$1^fE1?Kkmr-hZ+g-srHKbKbQ8qE?w;eWj;p4|1ewMg@t+WWHF zc{HpWPp_-52}hPMU86^whWXt>42Ls_n*7E4dB3mWQxJ2gRjS2nx3tLy8gc$E-QTa| z^J*lGG1Bg??8%2QU&gItcb*& zA4&uj`jrPEwMkW4Li=-$-j)2&XokhUg!qPZe=Wz-D#CE#x+I&2lcl+&6r0ima~<`= z+8TR2_T;MH4&l3DcbGo`>ai`;gmQ0cbF?lQSHPH0_Ljj%#oMA;Rdk=LnxyXpGFuK% z$e(k$Fa>Qy?_XvKPhMF!V!n#&%AQ?H9eF)(VhCByslHApRk=V$)7W~?uKSFhDU#}y>kYn-Pf;L(1-I|f*B zrs&-L`WV<0xaaa1rbRxjQy3Sklkrrtv~+QAtENKWjU~m^{Og(|-V>8rT{=ZaQ)-(E z)-p<-jUO7iB-Om^vJ%tu;|H+>W@==$Qg2OIX+>795}dGi$w$k)I*9X6>Hcl2k;ySn z`CdSg3-6h(>Sp3r+aKpursjq02YLGjDh2n9yB8<_Sbnv$_V*5J#J#BpN@Bj-uxE_m zL|zw}RV$%3r)P*CO^AR-Z?=r@w_{&M=DP_58p=SoZZiqrl9`y_)zDWv zIi}C)rD9H%oL-$@5jI z&(Qzl5aXY}&9MvH&b>7IJ==fm)&DGR`L9m@iH7`_O#kOqD##VG@zrVW?N&hiTd@KT zBxZm5V`*GD7M``Rz4})KZU0n7?buf6``a{iyxO@aMi)2a--@n{?Eil6Y5gOKjK9?S zd->nbeh0>4e#$luTtMO6zHdSRo=*Y)+Tr6J&JtQI=>_rzw?Nn>HS*m&%Fj)qH72#aeZ$yWi zSk1S^bx~Oasselg2lwxRvv3I3;%g<=yHQ_L?(-RW7PAvAKfX;~RWsO@7 z$1miPH~ft!7FN`f)WGJATiF9D@=~BUE{ucw35Jr%%`st7b+NS4E5-F14-Z^e!s}~D%@^b} z-q5G$OO7~xM%IS)2&t%LQ%hTJOs`~Nyz0^%Ga6#U?5~W?5*yC^f15jryuboMR3?*10V&9FR+01)-g&8hrKEzfGfOK=@ z`zUS@_EBjwm=jv6^?n^&WmguIm|Hx@~~}<$lsr zmfjswp-bry^gEy7NCBoQnzK zCQWm%8x!v{-KuARx_B~suIJ|%8XB;2O3j`ZdFS9DZ+@gHY^dZXh&bfvxV}6|hRxx+ zWkb39*m&$h{l4+R85$Hse6vy#rW9|x#P+xD@$il7f=LQPT?GXaDhH&oGCLd8#9M=5 zpatO&my?BYwh^)PYxyH)(iZ1rFEWCqu@T#pXQ_jFDGKT8T^ClRr*M1eO8m%8cahJk7WnBhJgrqggHnE~>_x^9os^j3Zkwwo|;8q9sU0ILwk|k56i{ z%CRVhZeMq6rF6#{D$Qk-I#^TCU8oKDfx8q4Eig)LfN;NX8%ER}A8zD0#`rK+#L*ilpD8XrAtN~q3Kk{kx9Y@b zN%a)8W8n?Tl2r-l*~W#}!lE0|vTg;TpY#2>kBzTg-IQa%mKQ7v-7IcVPg55%1_SY2 zJ=%x_R8lPt;ZbfU~56qp!PUd~6Z=AV7Q3D$O{b6*<=wLrQ^XdRXYv68DWUOLbCTg;|q&sUQtu)e! zH3hxNoE=yR(Pd5H9CJ_X>sB_IhNcmP=*L!n%4dx=ZkpamB?Ax4#^B(#4a}DT#R%KB z;nK9R#+Lc=5NLuAHqLfE#$ddR$uc_Q?L^_#jGXe9VP2+f1eQ9P#Et zUTe2l&ONf)Gel9q(mEIV9)Xk901}g|S8=v3*)uW7G&gPkkW5taMN{y7M7nk>bohqSU7 zZ38cK4qXfM^V zzy#3{$O>T--r{bZ?Rg2%-!x%!mo)^W#*PSSLBBhT>3+X3Xl5xL+E&FEP$7ysefB}5 zNsU%Z`7g2LTP6jXA2%4QOZo3!V4uV2SbA%Np=x5{{3*$);YCWv(kRSAxN#i@5 zn0oN!)7BFyg%pkXbg7?JLEv}Hn-Bc2m{{n{Gkn{6r57vEkB`}PFl1c^5~1gkBe?JG z+N9kg@#^s}JEeU!xDR9QYw5WMSoM!neWsevQ5+4yYtpm!E zUnUIwmIh1n#9fB4<+N9Ox8?hJ_4v4uD(BI2Z<`4f=ab&L^5+C_Dn#Qu2G=@b4EHFK z^`%l9=%qKVC}Ehlyg_Ogq#wOEACN+q=j~1RzsT3P2 zijehEs#zgICkja7?evX5Y4`AT%oB;V_X7|I7!4~Hf6Ao^8*dxZS!s#7sASv=4xd-P z`vzyQXW|KqoR{t%Ql#jM2Op>0;bvNB+-&uCLi_=&KF(;3yUz9?qB#4P_}nG_4Ey2t z96m3Qx3W|dN|0WF-0sKd{+6zGV6^|?dbB2KRHhGVfrPT0V!S-m&}%f1+F?xB;^6U~ zP;kTpD2Ch_nI2p~_u!^EUW3+z!d{&*P>-x+1_g{==(Zj&qId4}ap29e%XkncvD4n^ z3>ltou6NOdZ2p{LzfV{-p`pyMt)@;5CN4a620>fzThdL?CeA=MQ8(rZKVmUOvm#cRg@wU6z(+-hGPDpvCRxe7D9OAl;#ED^_fr#!!$*vc5 zFTao7lP;_>Zu>T$SYvB6KumCz69$|^RWKfotv`It`C4?4RI;BGG-4qL7($Ya;$Z>` zNa+AyO_HBw&mH4%bbQyM?}09lG7y%S;Kfa+%99pP3H&-L1o#Mz6I&eI`s7R^6vQ~w zXjTc^5~q7UCPnM}9gm}%`?-+qokd|qwl0_Zw7wPQ@B|_aR7efOg+iwmfXFdX{|?utwKu*1i%WO6QNK3fl8TdmgQUd~UTZM?8PmjMjO7h# z^9{JtMvLGU82J4xru|rk?()?O)d0h%<9Ry|Drt%UyowpK+Z{$3zjS2I-6%&t)@i+z z=AlcH1Xoq~7dHMpMeA_1zv}7g@H8uqy*hpB{>}tV^{Zh^;ACIp#Pm2jBLbHQ0k;T} zF{2zErRSYsqWra*Ag@8hCH`scxGk&!lOiMXpzv@j4sfOJj60R>-hHOM4%Xw-jWkiY znL2B6rqW#=(t+H6J}p1yh;Ch5WGbF7t8e-KO`qT#GYU7*Ey+lnh^EaKX}FbPx_<)I z_EvOwMwX0uri2MLFx6z~#w0(90$Wwmi16=hzMtRHj6b1?B);R@y6-o1LnaV0V8w8} z+?%O<;y8Ovz8_Sm6oGr999H!C)HAr=u3?AqoOQX^M+ZXqbR^$(cq%^dA$SV{osbS? zIIQlsnd&vS%jLcg+@zmgkLz+ncN#wNeOFd}MM(i+7>GuY3K4R;!KMfBm3b0lN!K>H z*sEy(mU*1F;(82w>TSLDXt_0VJR1x_+13=YRNG1$VOk1Ka>p{e%||Z9Ub+tS&PRuC zPDlGsHFtJPyGEO~!ell!ZyEo7m2}U0GQP88$wsa{_d(O#Vog7Gaos-7_UTv}o@JVI z>($Mj!Hvf~i{4qDFMoOXd076-;Q@J9sVc+V#V^xgjw3DIEI9PeL3Q_hnQ%)jr4@h1 zhcb1kpFyc>M7DbZD{EzduBJXbRQ_y+O@oS%momvPOu5g;2TxYSsz0kMKqFH3Z#wqr z3=nN7=L{0Re_;*~#v=WsfGyoVX06c(YGPl^o~PvMRkxQj(N@f(*e^GKPkVC4+~P)u z{lj{@eyX%Z+cDCYM>>WxH6XPBWbDLo6Sh7`?ZKoChM+TjYOnCvQe2ND?!Bkp%?d+4 z%Nssnd^+l@9w|PCGR%7Y5Yz@7|7(})fR%`zBM2soq7+f=ao97;?dDb?1W<247#%o- zmd5JtF`V~Okv&9yO#ixtp&+ElLDJDr#-1fa$2 zZaW^1xNN?K1bH|}1NiyO5);NDJhohb6EgM{oidX1+Ao=Vl3Qyy;L4jrsT&Vl#b;XB z>KI)}Fuy2S*r)6)-zPCN)+h={r{bqrIYtGNjKlvnw@T_lc}OkQC%oJF&&*D3 zyoe2%PS04%v$xLOnE-=+oTY!`SoYJh^c_kB0U+6MhCgOf0Hn6hUiJ|~KO8di9qe$x zvZ7xyxgzgJLP*K&6v-Y!4eu`9f$n;GJ$c*>?PGVty&D~Hlf6G=P_ZTf zIT_jF+!vhkLHt2@F%aAs80Dsp7TAbfp%pCBnKR^BqxjG3GlbfgdG>_Ls12fK@B6#D;l+G zBreDiWT47P1uyGP?;g7Lsz|u|2a=NY%0N)P_JTCp9;x_9?rz(Uc_x2cmrI;jgc_51 zG=G#rt)92mzu<`YY8t!ot+U%BDvQ>5t3`&rYE}&vR;_VGLJ%8BPOV=vUd)Q?Ssl>+ zpRVk)agb@b})Kri>JJo*`IfFEd&1T>W;s*jek?j_GZ4_BK<% zv%#@2TK^7-(gTEKi8c=9r&lv|x_G9v8AgzgVHLQu%sc-B&+3*mLBYK#A~=$9tG96H zFZVs6ikT&K5XAssAjHhk1SJd*cam_B8i^l>7X%=i4NX3@oR1G~XU zp{w{T=(WKco}17BYp1+rVuxcSnCqJ}L6^%QN~QRf7cjkrH88dSz8hlJXy|65aiDW! zyR~8{4+X6an?2VdK`BVU6>x{OcDhmAlEoeg2#{1!tJ%eCD*}k>rMQal#2?#@|L){N zTnO1u5-Bb7(MieHB87es@_dcc57BN*;q2FkT)qP9EA8?@Zt9{}w^*j-5sk6Q-d3z1?vOhhr@sry)# zewfqJE~?Xa4l{3R@aM^#9j5t>m6w&K${Q$!K2Wn&g9v=`=_>?;{N^%!mXTVk2ok_} zUNPJg+QyjgHmwGunnR^wYj~?TTj?{y;gr1eyP2QspFD?9n;b!f@Z;(1oScK}A`BTJ z%XE-8qi!hAUMY*C@aiMS2z7IZ;9p(mXQ;pPk#XJ^IYi+-y29XBYVda#l6Us}3=DlWF6u;GY|cEf zk?fq5IN^%~49W{>4~_L2A$mHxdKC?Z!5@tD8~JZPEF#zVn6%6Ih@ zL*tqviIy?eJwjc0;xW<`PQksKX($!S08Wzq5nR#QCFY>I`?&-~eNa9}?#7gl^}c(V z3)Ux%_Bb;p&5>NxG}h)PT)TMjCl`p4sVw_&Tl-$#4*I8Z$SI!ZyF-5|?&9igBKOMe zKWwWyfq#%A7iSBR1L%}3CT5gt2877vT>Y!HlYiQQf+zs7o`iU}^;@#cHRUX>*#EDO m`LC6O{*TVjmWxW5z1vo8H+GrL=Kd^`$J)~NRH^yZxc>$oL?jUa diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png new file mode 100644 index 0000000000000000000000000000000000000000..d5ebf3e1e7c7ba540bbd3e3368d6e9119baaf6af GIT binary patch literal 15091 zcmeIZcT|&I*EbkN1VNhgX2b$0(nJ&tMM0!@0YxB2P(-9ds7X|s6e(gs5F%X^NdW03 zbi@KuB%y>Zp@f7UNJuil`+n}{d1uy|HS7DnH8X4G{UaymI=Rj{*R{{yzq9u~zr08p09 zy$fMyT_5r_x)}ff@V4!J_I3HZcLM+}JvF|1<$8$AGX7iAEDlCk4S62QhT7flGJBt3 zzQ6J(_j#honTz9xRU~s#63R>(-;aqMyyt#*>#nU)i)4$McXeG>W%0FEwMT=ZJjaU1 zddh=dleeC<1u&74-IIsPkH4AV^KEuZ(b96en$1Ls+2QM%yS+@}{qqvfuI2SrjX;^J zKxPi&cK7pBrmxQ4EFYWpr;wQpGL>WR%8k|hO;ITbm(%yB$LNDm`Y48e%P{ZHb=9vb z8Wur!ptJqy%ewow$m@E<&TB-UE_$!;UbDAfEscO+xYHv&ODcLUE&VBOIZn_Abm?jE z$A3-h*6p8QDh>UQ$J$P2;>^8t!^o2O?(R5ma)iye+yi7o9);c+u6xFZoVgTWgXq}G zBSltY`d}+elhMCAYj0H1mVFQ(5n^EYa4eT#U?6P^oD7s}weZ3yJL7h08ekzic_f&2 zp-&*0dH*ZpU#}TkHAbJii0RW!RKoZ?`PMxUTo)WT8CpbJdWgYPjS{z#S-sB@DZj(< z7sqsbFcYw~QXbHRWzXwuEO6b0QZUXa;JK7jyIuphn{i056l0m4ft?6-NX!E+$t{Hb zB}~&-nKu^EY6UwfKVy&V1mFnYNR1~WYPOyQn{)2`@X--~$b0*NtGaKW938NvF7NCN zhBkekI_)j{{k;wzS5c_zDFtTIR0X)B{TsnS%w$U=p-PkLNufKqfpq&WSSzkwCLQvtO*M}xws&*qBe>R64fXv!Wp2aFKsHIE9#0#QsO?O+( zIXPu_-(z*Eugoof?5&s?^HjRe8oAzCN)F7u8~}`QN}y6kCrIOTy-)Ri!Wq)_;*p~h zpQluR{#ma2UN6cwFr!@3b0XMMpKE#fTknPr(s=~!iVdQDEb8+tMHV51snD)&?AjKc zZv8>jOsHT}=7&j>-;0G*qWi1q@b@&nso90t6Ei=*_0}m!dY9@*IwD43 z*B1yUM8}#oQ(^QIpo4-`uyv5rzSF5{PVxK%agmCXnveWM185Dcqd9Uh%h~ zkm)wDSz+A!u0=-&)^qdvAP5{=>f-mP{2S6V^;Z+))xfWE29`7!(ZML15%g_;?*J)p zWJ@Qd7~9``qLTq=!1I1b}Ztj zuGWj+vCOuZET*;>;%`&`O?M$N7-Xgk^WCdbuTZ+O7j9>UsqTgNd)?jXa1_}E1ON12 z;37CUxadC{Y+|MVn(FhXD*OZ;KN0j7*4)44RgY?-nlPLHh8nlpj^897=9bSU-`_T9 z`s$X`B3l37;Id$e7-Z=mfdsMN@YM}_~)^^=3 z|NKRlv#=y%7jUq-3;|4OPr%l${eiCh^!in* zZ1u+`r8>`GAk~wqYm8IHTJv$P25p%)0NY$ET4oBwkQzQx(Ef19kSU=A>=ThlQn(a* zgnVMcmVPsI2t!~vE+Q2G6d;6y;|P~&@%xZ_HCk4BHiuUxNE?(5gt&`vB>HZ6@y3skVU=Q$s+hZa&Z@*`}js2IB4Jo>U^`mo-&nl*{5qw9vKZ#tHjid5(xI9x{ zzQ`vIjVLwf&{e|qX(v(Nn@)&gCsuHSwH>fu?IBnBQ&CF+cH`e}&xZ|A`b!9ulA>xs z=%d)D=*Z}r6oGgPiL3p&XKu--(8F@%e{s2m@pd#qSId2gIL)ywn@q8zF&&oPoovyX z!GU~q-1-x*QfJ}qh+AhGfJzY^T_r&1cX%7t6t_<$2_4yn1{)i&%ZC0CwgszHpvsQu z+>d%P$L;U(FX)Hd$nVVoWyqBHIcanqXdP-^J7hLG<6EZQYIZZ$7R~iux*#%sPL`SlzcC@l{4Lj{q?HFn- zjkowM<`DHf<@l{n^1lf`Q87hk^Vh!1v$oZFwA-D zjVQ6^{iv!?WbR=li1Q|JBK9keSo?d^VCuaHw0u2&M%Ec6W;rF5GZB)7X45*$(utJQ zVlw>s9s1su@ormTz3iz?^5#25glh{@RW-v+xp{KgCwR0167Gyx1~zKj&MUX8KfN=a zkmP{PWxIU(s@P5IYV0j3OucB8b3GqX&*r^lD zK_3CA>E2DR5Imh(7?9J=vods7tRey)}G5iK6g1^`@m_CHsaOwuV6MJz}h(epYC z;keQA0(=e7%-{qyrye5$n$P`51(z~Pnx4%EEsS=U(w;-H&bs!jO&*I{HZ^?8dWdhw zh`06iaGIL6^VqjEF(|3-n_<0oHTUvvG(OOG&Z)>D7)s{}@f!!vga)TEB92|Oq=s+L zrfBc4vE&5d_{E{rt?wOKdbrR~fvo}as1w!8!%N0o z?=>a@2?^}kJTX}gukj0{&-D>^?GJ&P+pghSVopN3Vlz(?8=6Qxx%7ttQ}Wz;RoXhH zN!(MXc`sFIH<5a^oVV|u<;bl6I1tUXl^UI;Z^RO3>@##Ej~|4PTOah;fFmgtRpzwq zH`N|1O$)phbz(jEu@k}@pY0Hp$-?hDwDWD2&?BYk;Nq0qO>^;W@yl)A2~VQ9M3Q^z z0~(##pPVGlPg*15^O4WD;Rf%-q|!iL1-$-Yv^-M;<{*+d59cO)bjdrE;LqIXPU21S zMz(77bf`D=@pDcdV+kW=lpji6`@O$=czx6a=SLfT6(h}5PbXrV?jUF9Rcym;2CFwd zmdf-gyI!BKUy#?bA*7;`ji0nmray-XYbKr6jFq} zt-Kdb@|_|rog6U3`Jr}-l=>W8?!X@4{6;&lHv$sr=Q4h}a{J|Q8d_co>?0-Ac^AP_ z#Q8wO@KYCleu2|9IlA1~UJ`GS!JOjL?y!d43}cD9PPelj^YNY5n$`9-vH%JnghTRq z>=C=MG3-qbT=xhOEjTn|zXQ?}X1ul*9TZMZq(moIb>b{mB}Au{7kwy7`r#KZ8RCXh$m*2-!DE6q%x;v8x<^T$;?eRS z6=JC#gZC)dhRN|SqYr+0o5^1_V9*G6bsk>Ry_#!Ug z3G6&Y?W1*b92F=Q`+S3K5`LTrFg(t<8#s0B^NLWOd&0_&;N~xTNWCJXZZo>Ug(w_Q z>Km~}0iTq-9f*O=*QU1I_chy+vh2y5p^ML^$3Jd6I&cvf5mRbCrdx@ip6`k{bwByj zGkqB%AoDVAUs5~Ee#v49E0gK{m)QYJtY*U}e;Mb0RPLZyP3ehfHmO6cynwTjdzH;^ zMtgC>-4QWX699Mqf3!qDvDBr1rcFlj?Ro6b$-+6TYLxVC_C?Vn0N)VXIR}=Y_y05N zUzNyluV6p_{w$%94%7N?kH@ri2k6<_m1j$@8DX!L^F z0QaM$Hx?m=mHGz-TP>V7U$6eU8$81CcHpCLFR9dG#y8+8IhR~C>Gh=H2L!D8kl%$?~aF&6idSi9n|cm_3C;FnfAgu(Hd& zJ5=x8X&2iGy_8}w_A~9A`OfRU2v`uKwRGqyIxXe|U1+|q0GKdNlI3L9?Pi=kkbNI4 zUNg9JG%J_@n|w}VYnEuDG?JrhyDw}y@<2Em!*FqJF?dU%)$OIX{9F-MzEgQeT%3$n zAcYg(v{UgXQvOhzpgtQR_ zfj+I)zZK4QunRNoZoqj((bxG;#}@T#&P*POe$z)g_u~PiNW>;L;Ek`~vRveI4IrT5 z&_BbvXcMnS!{5En^tE1emnXK5HpmmBT$Nl#7O#o%u;-f{fluuxId5Ra*F5%UQLYV!6tJaREog4_RF1CGKxY^J&Bn= zaY)cl#h54wE%zkabxM_j8hgrEvR{6!%A;(tLDhj6{v36>FCJcgLI4Vo+P8VtomiZt zhr6OIM7BwrYGseEsYSh9q}S;5qitbljL+l6I*PQswRgUr#XH2xS6krr`6ly2e@ZIP**N0Xt$g1j z!j8^bI-I%3!l0%?J)^izu~O8y;m$aZIOFaZLHA7h?zayL0`~{)g&k+1qUluaK$S&s zuw~#htV8n<2tX2GxuNoWBC$4yjV0B7zZL+Xby+RIK!8 z+`|)D1weo*;xp@s6mFOAR}oz|1pSxhkZOEKw{h_Fr~&iqnm6*zx^u&4<+DA`1Q`@+ zkid$mwoB^aUJkF<)^b6iB0Y5s`rDXpu4bq2z-3NQ#+mPhMx9k^Z48PIiBhV!t)=%B(51s}yS(+;8aV`F3C-R`Xhye@hE`kIkUO3&Hcy9HKcfW^Hm|f& zz>tIp)yo@Ohd}T#B@J8vEgYinz!_G>hfwrDdgHSy7zf(H!51tpmah6yg9Lb;G1T;j zZyP@+6Jf?(spFVXPSATb$lPx9AuSJFZD?-Q@XPEvEF5Sz8C-<_tx!RICjeD>gg)^i zd%Q+(scNV7qfn%we1Bbmf%+k}n_r$=21UqR`0-Y50psIumeyu*o)=p*EXQ@&MC8mg zeSNbGoksF8pj4dinRUF@Y5`Y%k)Pa3d)|Z{CL9mtE7F*Ed%0^oE~@9e?FKn-a9zaG zPGiXu&^izuQom>@a=k2D#}2J{1_yO2?QZqrs*UhZ)vA=Yp&^4`pSiuj7+YPtF$0SWkI8|jyv$vUTNWQ=)Q$J9cl>zHRbUv^0*>Ch(q(}DJF#@y0ytrU#hu5BDyJgzkU(NWijA5z3p z^3y^;Hq@C#;b0;CV`p>>e7h_UgUU4wXT*eK_v>0J^nZ{ z;f3ETlEZl@kP}}M<^c?kl9mfc$$gPk0JYSKh-%#W?rfS6Jyw``_B?$%5HJr;ZjC(~ zlUobB+S1#d`yqd6>uul8;mxoJuX|FZXg78Rn|i*f-KmBFX33X)=0*Zr9(NFmH?0_g4E6%icj2Og6w{ zHfs85j?c$w#K!kgFUoo+eQU&P_!N#4Szva*x(QwQLs$!Vy!=&E;%6QqyHmU(Gcq}0 z*=NV{75P9O0Ub17NY@2{Yg)jo&*#X!+Zr>T>Xft6^|vfDF;#tNTgfiTu<|_FIlcEO z8?)oApgiOkceihr$PJ1CpZbhQc|%7I;g?7Z2F=8^IaVe! z9-bPZRK6o1QO>O?MxB@67mhkGtRG-=1-j;S85_-aA;?{{f0v*uv| zf|YRR7=GIqdaEkYwnrsnW$Q7XH&f4{kS_QQ`bmh~!YvV*(1#MwueIr0%NPDe4^-r+ zyv2%Cr=rvFdHI}4$rl?5wzK4f2UCs37-f?P#QJi=dpsCqADSF!Brw%l5-T#%yWtWw zlN+#w4H*CjQ(dmY5-7Ll)r@eVVw@mB0q79a6xX8d77yF(mQ-Qe+unMvMrBnBKz*6> za-4>Y7p!3DUz^^?XX{|ac$D)aHVEUsA#R_2vm5(urMYD2o*Qmv51702f7*F`u z8K~}<9-R^(3kF z-^+ge+`RlB@A;24n57<+j^6W8Id1G@`OK;wKc~nGuNP-0V^B{kR0KM0szg0tkCR#Y57RFfytn;5*SIXsdY=Gz*vfrb4A!hYY;GoL>Xo{82mjbweEwEzmu(H@OaqLr;=j+^MqnP zhQ}+Jxr6rOXzDvi7wHuRAK9~S=9#F7D5#GDH zV!5Hd422!j8V({T6HPFbTJ39%rBu#QsmkOL=w2R0*{&AGz1(*fferuT^DAw8hEA8} zjw6ADg!KmH&5SX_`Xwdca^1j3x-OPFzO8DW-=)HU6H)$_Thib-xKCk2??5}U(tdqH zcg&I&la*8WBqi+JXTH`R08l8~d0sDgZTr%jpZeCAD%zqfV;0yNb0!5EeqDaxwiOo_ zNqsa|j=S_{?A()QplYA6-2w(pZCmcfhmPooT?w0lfc%;7kycvc`OMd%cQUn_BIuCs z)6x}>J6N26E|s~FaPgj($*}MF=rp10&M&xSj$5(o3#r+S5gCiP0r)NZ^IpCTXlL#=WT=M+REyXOV5*miqCY)T3sHL6thY>eopD6W?Jfqd|yEADM8?v%fz7KFUw7i^hM9Dp?=XubMh5WsES1aQ0?u?Y2`!O~y0f);qZ^YZ)u8&}FZ2VJxy=+t}q>JPe-L1g!F5>YVI)vl`cIU+ZoS-nx|A>lo;C zY3nQOXV(`>OGLIgl0-0tIa4*5H_u@oP}3;Q{@4)V@#>Jb=iX(k=g)LpDvZg}rRM4O zGfg~8R@H|dyLDVpK!5#oI)%kAc$%R{rmjMa$~q=Cyl5*eGA@>FrqB5Js#u1!11*T) zfgnS|y}znI4R>g$K2-`Hm5t2`(Hftm{Yk{I6RE>%lpoIZ(zgOj(k3{R>kroY_&3HkOhi+D4vbFI)pLK zCaiFNOlRp$O3>n1rLWZN_0alH@JsYl@82dt-^s}TUt}; zJGtOM?No|jxtCu~+O_ytc@1m%`Qk&mLbi7DW?-t5$LdP7<+82v2ZK6g+ZPe40@3G{ zGs-b~sRLsgCNm4Yb~Rfy+wtr9*pUI(MX*8)AiM;D7=zx7c=>)}SI5BBBzd`mY&muG zn`^meN>_UL4SM2xPx%8NZJkO*k5c43%#>6OD$?!o`RJ7J+&NMzcuHb}QMn|_HM&c% z5}?2LJd?!pQH%?;NC+MemE%&e9c8hDHc8bO`lmmuog#Wof%{-}lTpzR>7|6Qp?;mt zp|n>~QJ)JSUFUzfk02%z8pl8XVjIYYbdix&T5n%eIglYTmly51j2Mlk8RQcf<%DSI z7NM*f{)3y-IZ50dGCb_}?!1mxpnm&$^RX3S0Xu*>pydl!Rxnuw+b6}}E7M(fygnaR zs8Z~;HjZ0rSVS3+kkN_hU(gmVwOk6l;KA@kt6^RR?=DvWFx?V zftXx?P<*Tszdz9HNV0IX&cBoH3Y=-hHQLd4>S_i5MwCtY-lX`(^i9eE0N#rdIP*SrU$dW^CdwbGw zh&jevyNNFsNgV&Wid!<*H6&3hT-iaS5U{2WNVd{)4fG`FWor_R@4%}k-(M6`)8o+t zvsGeJ`FHv{B2OUtVCbKU+3oc9w_a%m2EtO-A$Dc!(S23~F{Ovbt47@K?%vK1ii}O@)^3*EKM~~1Z9i>$ISB0rTU0*3}N0%6< zoQ;nsq`iRmCC4jqM)N<0dRXb{9Ykv1mHBX${c9Nv{kdOCgRt~*f=Qo1ZlF*$QCYH4 z(q;ifh@q~y5hE&zO@^3wLxcE5?n;R3_=c1b%a8MIfONqOE~~m6X8CUZkgkY=PTMS3 zmUnmbkt?E{4PeST+v5X3IGXiY)S;t*%>)aUhssym$cj$^0JO*dUFB_d1mmJ}@a7X1 z?{CZJ0ps#shFnG6n%;tp_(i8McMyAtb8_Jv-+s$KKW)mXsYCWPOFdv%t#i(?LT+x> z=l+#Hk+2BsRY*m6l7`92y#xtI9w{3cBqt|Vx0gWo9{!dvnyW8e(% z;Z@{n8A$`%Q@4Z-Ydh(4me^27v);&<5IUM81X)Z^3EoW-F7S!9d) zZw}DS`A-{1`QSbFxn1aiDL2Y||92qAW!x}tEAPSX%g)k94ssgxN`5wf?#9 zvdCFbjB)fiX_SU{4OsZ-*RBP6d2T^`a5zq*37s|(yGW~g-HEo1;Pa0}R#e(`X^|Uk z$sMYHvL3e|adUJJyqHxu3RG!dVvu))o^&!GUD;r*)91RPj@=nbn+abX$ft_24&{gv zc+JAt%xwwEsJKUP@-7HHVC3;!!7@0R1S@ zxLX3^UL1Z-5W*oGB|UgIdLlZyW`&SbusgDgDqAEZQ&Yg1r+!4eY29AdT!hFaj$RZf zER>It-oZInI<9^a(s4$|%KE4;IB?fxY|rS}ZEGU>Fr~?=rt2Bn4%aW632Z_` zhHvP|NB7GJAKXTC1xa{ylPWi+#vYn_|DEPP7re+6h z)%9n;uRyYKl5^b9rr1}cz(v{M-U0r1E@||_C4HHp_YplrLQAsoRvui0e5w9R0-u<0$_w>mic)@-|& zpSRH3;GIO$;7)wBvhsnxP*}P=@s^^BE4>hMMMP8bD0MxXLL1kzrfs4Y)`h5>GiveC zKmGSzQT|BLj(Y5?Jn-aA67w?4ghsQ1-+ytZUvtvECq9wfs%>UB;gCnwl+^HEsW@G% z^L!#9hc;=~n>9Xjq%t|Iw+$YZwS0{D?if7*Q??Ojrb*mrACRd;abJXBw!xQJFxZkS zLzMTJJT$sCQ-KzIcACdH=OxMX$8B66S9HIcps(Yxz*GhEaccqdUtCT*K|x8MYu;lX zz7@tkcr3njQR(tr#u!Yc>%;oOGF{VxhQwDEmwoYjF~nMe?nfy{U&oWLII8@vlR#;? zHz@TMx6hbTf3+x<7`lFzX$EEzCDmM2kXgF{`Ls`P`!M52iHBi5ACMg3zz@jH7ZL%u z;1wN>NMwh{71OM)Wy3!h_i756a+o76_81@N-t;eof2E>=9{g5q%A4F&AKR{b?9Jk5 z4}^j}9z)q>6~wlJhQI5uq&(F!CaMMxPu>kNFXNptCPDn2^(it4h_j zpAuX<8r8@Sp|_w;)TYIskm}8u+67vCsiJrQ?q2B+47e0IqD{O-9lwS&P@M}!s-0j& z5N2ME=B^&b$F{nwNa}yKv6f;)!2BK6 zP0QaY<&z;u9(YexG`d1MO8TrEm8Sf`oEffk>v^=a`eQhKY0kZJ%pG=GuybXGd50qfkFhi$=f5)B|GRDW=ac^)nB_<8c^v;OihrE1|6hvY`|8jw zpM_KZSXXnC(@z#HW$!F%&y==gB3T5SJ*RU|Syb#K0)_rZfxF!fVXe6=#tSSnLw8W?$gIv1e)_kRl>B%|sWEsM zU@_`w+1eL_m$%+%ZcP`s!1DeJr>;Ee54{7_n6X*B|54smTy1jhe#9S|oaOH%3WUHo z2f5p+jV2l3LW730w)PnwOkvSyBiWDQv3aJF87!(%RV+GyJQjpaHh+u>5z#Gp8ds!y z4tMp6vHIru{9U&$_I(4>mj$v@oD$X(W0Tr87f6x!wj z^A-N|)LBP9QRu8@8Z3;Ym4o!_0AG{Ti100r*SFqKf6aYGDIK?Yf1)3I<3txvY}Rb| z!0Zy>k;m*nR!$L1UkNVV3E_?nFB@*lu?Q)}y}qOooz<4I(zetE7P?DK-ypk!?1Z4vl=-|b}KCsM!l=e9H z4BHh?7AKca!tB-jfR89duCdAuMrP3sGqu|+YPx2=ViNtD-^Z~fti`k??S`3MDANUe zwy_NC%wCj_$_j56o^}ji9}(WLN=dU)5rIamipaD{yVAx$>1X@g!<-w>mu!cZxwB@d z2*1e%qR;oE%I4nF&lTph#<^fPETMqOpM!!~Wq;mt-1qTG=2uz8dB{Y5?Xr^Fd?D4e z)dD3_($BA|y*HjV(aq})Zk2M*D-u!!9p7z~u%X`_Y?0aFGK$Si1qjl!vf5&^=B2i3 z?&xM{UZ^_=xN!(VXfn!va@EAryuVpcnxOaO!Ebb4JG++BSU_u%KTc;IEeNg-(y)v(yq)F!sOE>h|j+0ywy zCg<)oweDYu;fFptq53a^-G5N${^8L5Q~7@jnNE@igjf|tTw$Tv9#PjvlC>89*Ur#l eg78d^MYgf1;rw(f)*q$-#s(HwOZ9F)`QHF}_P@gb literal 0 HcmV?d00001 From d8675602da0e52f7eac3d55776b7422ce1b11697 Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 17 Jun 2019 15:57:05 +0000 Subject: [PATCH 02/40] removing draft --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 1 - .../automations_monitoring.md | 1 - .../conclusions_and_cleanup.md | 1 - .../emr_instance_fleets.md | 1 - .../emr_uniform_groups.md | 1 - .../fleet_config_options.md | 1 - .../launching_emr_cluster-1.md | 1 - .../launching_emr_cluster-2.md | 1 - .../launching_emr_cluster-3.md | 1 - .../prerequisites_notes.md | 1 - .../right_sizing_executors.md | 1 - .../runtime_metrics.md | 1 - .../selecting_instance_types.md | 1 - .../spot_savings_summary.md | 1 - .../tracking_spot_interruptions.md | 1 - .../verifying_results.md | 1 - .../visualizing_costs.md | 1 - 17 files changed, 17 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index a353d30c..6f556bf4 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -2,7 +2,6 @@ title: "Running Spark apps with EMR on Spot Instances" date: 2019-01-24T09:05:54Z weight: 60 -draft: true pre: "⁃ " --- diff --git a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md index 2f835496..b29875d8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md @@ -1,7 +1,6 @@ --- title: "Automations and monitoring" weight: 110 -draft: true --- When adopting EMR into your analytics flows and data processing pipelines, you will want to launch EMR clusters and run jobs in a programmatic manner. There are many ways to do so with AWS SDKs that can run in different environments like Lambda Functions, invoked by AWS Data Pipeline or AWS Step Functions, with third party tools like Apache Airflow, and more. \ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 0d69df3d..0af270a8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -1,7 +1,6 @@ --- title: "Conclusions and cleanup" weight: 150 -draft: true --- **Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index cf7508f8..03e14768 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -1,7 +1,6 @@ --- title: "EMR Instance Fleets" weight: 30 -draft: true --- With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 795def87..722e81bd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -1,7 +1,6 @@ --- title: "EMR Uniform Groups" weight: 20 -draft: true --- When using the EMR console to create a cluster via the quick settings, default advanced settings, or with the AWS CLI - Amazon EMR will provision your EMR cluster with a configuration option called "Uniform Instance Groups". diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index df0384b8..f77e3d7f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -1,7 +1,6 @@ --- title: "Instance Fleet configuration options" weight: 85 -draft: true --- While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index ff5972e3..9815458f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Step 1" weight: 60 -draft: true --- In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index e1e4908b..5b9fa97f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Step 2" weight: 70 -draft: true --- Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC where you want to run the cluster, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md index c75d1a9c..f5e4d283 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Steps 3&4" weight: 80 -draft: true --- Under "**Tags**", tag your instance with a recognizable Name tag so that you'll be able to see it later in the cost reports. For example:\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index 6441516c..a86c236a 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -1,7 +1,6 @@ --- title: "Prerequisites and initial steps" weight: 10 -draft: true --- #### General requirements and notes:\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 043aec6e..8b33ebd6 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -1,7 +1,6 @@ --- title: "Right sizing Spark executors" weight: 40 -draft: true --- Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md index 1de1591c..6baffb1d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md @@ -1,7 +1,6 @@ --- title: "Looking at run time metrics" weight: 120 -draft: true --- In this section we will look at the utilization of our instances while the job is being run. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 967c8dfd..a72fabb7 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -1,7 +1,6 @@ --- title: "Selecting instance types" weight: 50 -draft: true --- Let's use the data points we have in order to select the EC2 Instance Types that will be used in our EMR cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index da658e88..8e50d12f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -1,7 +1,6 @@ --- title: "Spot savings summary" weight: 90 -draft: true --- When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index 3182f572..c39e8e9c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -1,7 +1,6 @@ --- title: "Tracking Spot interruptions" weight: 100 -draft: true --- Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index 7e56363c..7258e009 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -1,7 +1,6 @@ --- title: "Verifying the app's results" weight: 140 -draft: true --- In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index 0f79768d..b9fbc86f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -1,7 +1,6 @@ --- title: "Visualizing costs" weight: 130 -draft: true --- In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. From 3c0e0e4c6498fd8555a05cb8a1480fe98653f68e Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 18 Jun 2019 08:56:00 +0000 Subject: [PATCH 03/40] small comment on right sizing execs --- .../right_sizing_executors.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 8b33ebd6..1d8ef2ef 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -5,6 +5,10 @@ weight: 40 Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +{{% notice note %}} +**Remember!** you might be able to achieve greater utilization and optimization when using a single EC2 instance type, but when adopting Spot Instances the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. +{{% /notice %}} + Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=90G —executor-cores=15**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4" From e483b05ad50c8dd9203defa6433c5e4ddf23cf19 Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 18 Jun 2019 18:13:35 +0000 Subject: [PATCH 04/40] submitting workshop for review --- .../_index.md | 4 +- .../conclusions_and_cleanup.md | 2 +- .../emr_instance_fleets.md | 4 ++ .../emr_uniform_groups.md | 8 +++- .../examining_cluster.md | 45 ++++++++++++++++++ .../fleet_config_options.md | 12 +++-- .../launching_emr_cluster-1.md | 16 ++++--- .../launching_emr_cluster-2.md | 2 +- .../right_sizing_executors.md | 13 ++--- .../runtime_metrics.md | 6 --- .../selecting_instance_types.md | 3 +- .../spot_savings_summary.md | 2 +- .../tracking_spot_interruptions.md | 10 ++-- .../verifying_results.md | 29 +++++------ .../visualizing_costs.md | 21 ++++++-- .../costexplorer1.png | Bin 0 -> 31710 bytes 16 files changed, 121 insertions(+), 56 deletions(-) create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md delete mode 100644 content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md create mode 100644 static/images/running-emr-spark-apps-on-spot/costexplorer1.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 6f556bf4..86cc426a 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -5,7 +5,7 @@ weight: 60 pre: "⁃ " --- -## This workshop is still in draft! ping ranshein@amazon.com for any concerns. +## This workshop is still under construction. ping ranshein@amazon.com if you have any concerns. ## Overview @@ -28,4 +28,4 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in details during this workshop. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 0af270a8..84e98dcd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -3,7 +3,7 @@ title: "Conclusions and cleanup" weight: 150 --- -**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. +**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, you verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. #### Cleanup diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index 03e14768..76e64bb5 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -5,6 +5,10 @@ weight: 30 With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. +{{% notice info %}} +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets. +{{% /notice %}} + **When Amazon EMR launches the cluster, it looks across those subnets to find the instances and purchasing options you specify, and will select the Spot Instances with the lowest chance of getting interrupted, for the lowest cost.** diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 722e81bd..55ceef30 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -7,6 +7,10 @@ When using the EMR console to create a cluster via the quick settings, default a With the Uniform Groups configuration option, you select the Availability Zone in which you want to launch your EMR cluster, and one instance type per group (Master, Core, and optional multiple Task groups). -When adopting Spot Instances into your workload, it's required to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. +{{% notice info %}} +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to learn more about Master, Core and Task node types in EMR. +{{% /notice %}} -For that reason, we will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file +When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. + +We will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md new file mode 100644 index 00000000..38f9d167 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -0,0 +1,45 @@ +--- +title: "Examining the cluster" +weight: 95 +--- + +In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. + +{{% notice note %}} +Do not expect to see full utilization of vCPUs and Memory on the EC2 instances, the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +{{% /notice %}} + +### Using CloudWatch Metrics +EMR emits several useful metrics to CloudWatch metrics. You can use the AWS Management Console to look at the metrics in two ways:\ +1. In the EMR console, under the Monitoring tab in your Cluster's page\ +2. By browsing to the CloudWatch service, and under Metrics, searching for the name of your cluster (copy it from the EMR Management Conosle) and clicking **EMR > Job Flow Metrics** + +Some notable metrics: + +* AppsRunning - you should see 1 since we only submitted one step to the cluster.\ +* ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ +* MemoryAllocatedMB * MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ + +### Using Ganglia and YARN ResourceManager +The recommended approach to connect to the web interfaces running on our EMR cluster is to use SSH tunneling. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) to learn more about connecting to EMR interfaces.\ +For the purpose of this workshop, and since we started our EMR cluster in a VPC public subnet, we can allow access in the EC2 Security Group in order to reach the TCP ports on which the web interfaces are listening on. + +{{% notice warning %}} +Normally you would not run EMR in a public subnet and open TCP access to the master instance, this is just for educational purposes. +{{% /notice %}} + +To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ +1. In your EMR cluster page, in the AWS Management Console, go to the Summary tab\ +2. Click on the ID of the security under **Security groups for Master**\ +3. Check the Security Group with the name **ElasticMapReduce-master**\ +4. In the lower pane, click the **Inbound tab** and click the Edit button\ +5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ +6. Click **Save**. + +Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ +1. Click **Ganglia** to open the web interface.\ +2. Have a look around. Take notice of the heatmap (**Server Load Distribution**). Notable graphs are:\ +* **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ +* **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ +3. Go back to the Summary page and click the **Resource Manager** link.\ +4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index f77e3d7f..3af88678 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -1,5 +1,5 @@ --- -title: "Instance Fleet configuration options" +title: "Fleet configuration options" weight: 85 --- @@ -11,5 +11,11 @@ While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units -This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) are 64, then 16 * r4.xlarge instances will be launched by EMR for our Core fleet. -If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. \ No newline at end of file +This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 80, then 20 * r4.xlarge instances will be launched by EMR for our Core Instance Fleet. +If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. + +#### Defined duration +This option will allow you run your EMR Instance Fleet on Spot Blocks, which are uninterrupted Spot Instances, available for 1-6 hours, at a lower discount compared to Spot Instances. + +#### Provisioning timeout +You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the Fleet when looking to use Spot Instances, before you look into failing over to On-Demand. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 9815458f..1aff90ab 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,17 +3,18 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. To launch the cluster, follow these steps:\ 1. [Open the EMR console] (https://console.aws.amazon.com/elasticmapreduce/home) in the region where you are looking to launch your cluster.\ 1. Click "**Create Cluster**"\ 1. Click "**Go to advanced options**"\ -1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark**.\ -1. Under "**Add steps (Optional)**" -> Step type drop down menu select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ (make sure you have two '-' chars) -**Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: *aws s3 cp \ s3://\* +1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark** and **Ganglia** (we will use it later to monitor our cluster)\ +1. Under "**Add steps (Optional)**" -> Step type drop down menu, select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ + +* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: **--executor-memory 18G --executor-cores 4** (make sure you have two '-' signs)\ +* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** ```python import sys @@ -24,8 +25,9 @@ To launch the cluster, follow these steps:\ exit() ``` Then add the location of the file under the **Application location** field, i.e: s3://\/\\ -**Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ -**Action on failure**: Leave this on *Continue* and click **Add** to save the step. + +* **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ +* **Action on failure**: Leave this on *Continue* and click **Add** to save the step. ![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 5b9fa97f..ec53617b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -3,7 +3,7 @@ title: "Launch a cluster - Step 2" weight: 70 --- -Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC where you want to run the cluster, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. +Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC that you deployed using the CloudFormation template earlier in the workshop, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleetsnetwork.png) Let's discuss the right setup for each of our node types:\ #### **Master node**: diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 1d8ef2ef..2396e85f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -3,15 +3,15 @@ title: "Right sizing Spark executors" weight: 40 --- -Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +Building towards the running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. {{% notice note %}} -**Remember!** you might be able to achieve greater utilization and optimization when using a single EC2 instance type, but when adopting Spot Instances the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. +**Remember!** you might be able to achieve greater utilization and performance optimization when using a single EC2 instance type, but when adopting Spot Instances, the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. {{% /notice %}} -Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=90G —executor-cores=15**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. +Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. -If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4" +If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4". However, there are some more limitations in place. #### Amazon EMR default memory limits for Spark executors @@ -25,8 +25,9 @@ r5.xlarge: yarn.scheduler.maximum-allocation-mb 24576\ 2. With the Spark on YARN configuration option which was [introduced in EMR version 5.22] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew-history.html#emr-5220-whatsnew): spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting ) -So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types, as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. +So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio), as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) -Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file +Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md deleted file mode 100644 index 6baffb1d..00000000 --- a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "Looking at run time metrics" -weight: 120 ---- - -In this section we will look at the utilization of our instances while the job is being run. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index a72fabb7..34b1698e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -3,8 +3,7 @@ title: "Selecting instance types" weight: 50 --- -Let's use the data points we have in order to select the EC2 Instance Types that will be used in our EMR cluster. - +Let's use our newly acquired knowledge around Spark executor sizing in order to select the EC2 Instance Types that will be used in our EMR cluster.\ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to read more about the different node types. We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 8e50d12f..02b1bd40 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -1,6 +1,6 @@ --- title: "Spot savings summary" -weight: 90 +weight: 96 --- When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index c39e8e9c..b01eaa7e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -6,7 +6,7 @@ weight: 100 Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} -In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions, but when we get started with EMR jobs this could be useful. +In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful. {{% /notice %}} @@ -41,11 +41,13 @@ The only way to simulate a Spot Interruption Notification is to use Spot Fleet. 1. In the AWS console, go to EC2 -> Spot Requests -> click **Request Spot Instances** 1. Leave all the settings as-is and check the box next to "**Maintain target capacity**", then at the bottom click **Launch**. This will create a Spot Fleet with one instance and you will get a Success window. 1. Still in the Spot console, click the console refresh button until your fleet has started the one instance (when capacity changes to **1 of 1**). -1. With your fleet checked, click Actions -> Modify target capacity -> Change "New target capacity" to 0, leave Terminate instances checked -> Click Submit +1. With your fleet checked, click Actions -> **Modify target capacity** -> Change "**New target capacity**" to 0, leave Terminate instances checked -> Click **Submit** 1. Within a minute or two you should receive an SNS notification from the topic you created, with a JSON event that indicates that the Spot Instance in the fleet was interrupted. -``` + +```json {"version":"0","id":"6009a9f4-cc7a-8a77-46f2-310520b31e0f","detail-type":"EC2 Spot Instance Interruption Warning","source":"aws.ec2","account":"","time":"2019-05-27T04:52:57Z","region":"eu-west-1","resources":["arn:aws:ec2:eu-west-1b:instance/i-0481ef86f172b68d7"],"detail":{"instance-id":"i-0481ef86f172b68d7","instance-action":"terminate"}} ``` -1. Go ahead and terminate the fleet request itself by checking the fleet, click actions -> Cancel Spot request -> Confirm. + +Go ahead and terminate the fleet request itself by checking the fleet, click actions -> **Cancel Spot request** -> **Confirm**. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index 7258e009..44f80681 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -6,21 +6,18 @@ weight: 140 In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. 1. In the AWS Management Console, go to the [Athena service] (https://console.aws.amazon.com/athena/home) and verify that you are in the correct region where you ran your EMR cluster. -2. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: +1. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: -```sql -CREATE EXTERNAL TABLE EMRWorkshopresults( - words string, - count bigint) -stored as parquet -LOCATION - 's3:///results'; -``` -3. When the query completes and the result is **Query successful**, the table with your results has been created. -4. To look at some of the results, run this query: + CREATE EXTERNAL TABLE EMRWorkshopresults( + words string, + count bigint) + stored as parquet + LOCATION + 's3:///results'; -```sql -SELECT * -FROM "EMRWorkshopresults" -ORDER BY count DESC limit 100 -``` \ No newline at end of file +1. When the query completes and the result is **Query successful**, the table with your results has been created. +1. To look at some of the results, run this query: + + SELECT * + FROM "EMRWorkshopresults" + ORDER BY count DESC limit 100 diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index b9fbc86f..dd073980 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -1,6 +1,6 @@ --- title: "Visualizing costs" -weight: 130 +weight: 145 --- In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. @@ -8,13 +8,24 @@ In this section we will use AWS Cost explorer to look at the costs of our EMR cl It will take 24-48 hours for your usage to appear in Cost Explorer, so you can plan to come back to this step later to check the costs of running the workshop. If your organization administrator has not granted you access to Billing information, then you will not be able to access Cost Explorer, but you can look at the examples provided below. {{% /notice %}} -In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=Name, Value=EMRTransientCluster1. These tags can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. +In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=**Name**, Value=**EMRTransientCluster1**. This tag can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. ### Analyzing costs with AWS Cost Explorer -[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. Get started quickly by creating custom reports (including charts and tabular data) that analyze cost and usage data, both at a high level (e.g., total costs and usage across all accounts) and for highly-specific requests (e.g., m2.2xlarge costs within account Y that are tagged “project: secretProject”). Using AWS Cost Explorer, you can dive deeper into your cost and usage data to identify trends, pinpoint cost drivers, and detect anomalies. +[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. You can analyze cost and usage data, both at a high level (e.g. how much did I pay for EMR) and for highly-specific requests (e.g. Cost for a specific instance type in a specific account with a specific tag). Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ -2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**MyWorkshopEMRTransientCluster1**" -3. To +2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ +3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end. +4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia). +5. Group by **Usage Type** to get a breakdown of the costs + +![costexplorer](/images/running-emr-spark-apps-on-spot/costexplorer1.png) + +* EU-SpotUsage:r5.xlarge: This was the instance that I ran in the EMR Task Instance fleet and accured the biggest cost ($0.17)\ +* EU-BoxUsage:r5.xlarge: The EMR costs. [Click here] (https://aws.amazon.com/emr/pricing/) to learn more about EMR pricing. ($0.06)\ +* EU-EBS:VolumeUsage.gp2: EBS volumes that were attached to my EC2 Instances in the cluster - these got tagged automatically. ($0.03)\ +* EU-SpotUsage:r5a.xlarge & EU-SpotUsage:m4.xlarge: EC2 Spot price for the other instances in my cluster (Master and Core) ($0.02 combined)\ + +If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you Filter by Service=EMR and Group by Usage Type? \ No newline at end of file diff --git a/static/images/running-emr-spark-apps-on-spot/costexplorer1.png b/static/images/running-emr-spark-apps-on-spot/costexplorer1.png new file mode 100644 index 0000000000000000000000000000000000000000..b75fb95bd68bd7526290240901ef84d62497502e GIT binary patch literal 31710 zcmd?R2UJtr+BO zHIyv~0ck=Y2%!iegphVu*lv%W@4NS$^MC*S#~tHl3~{Zjxn_Om`#f)%d3r-%dmI0L zeh>(>?b_ALMj+5e90fs}^1$(DnxIKb~gyclIDqpx3p_7jF95%niH79H;w_(%}2x6TV+7pyzKKxcY}i z>v@3<&t14+i*Wl)wyz0+vj%~A&3vwUuQs~l`-Z91YOLz83Ak@sBNyly z@IeY?fHtgO;c^U-JnL7`UP;{g1*q_z7SGBjZE=%P|HU@}Y^LhriH+l$3-ey?V^a^( zFu}kSREDb9SL*YCEKQLt`VNFSOPd%bdDwo~0~JW4C$9ScDejB{sx3Gieb_1+bY4mZ zhvBT1(Kp2HEM`}BVLx^~8vFL6o`m~pyr8N)5Tj%PG0Ggf_J;*xXbLhu_2K@qCeTN# zO9yp;mXHVe_8$JI4LP<*fvQ-U^MY=S?;lsGhw+0ZByoQn)?7RC6t%fwtbR8Lxtp-F zc~^lk=#Vzgl|L3giCklz+q+#P0R)OustR2l7s2*_+^?S%MivCM7$SFF3|iLO4c-yj z5ON#@%G4`3EQ#|G`StA-*16UE8OqdDZ`JcIM8kfpjFTn`4ae89>*UQ)Ff@mhm^*s1 zNh7GTa3kndY3td?a=2yk+#9sPI5nhQSrd}R8F!yt(Cc_^U9mn?I)9-C2gezP(DQ=5 z()t>vN_rC!2HV8L8OQr|(Nmn7-bBtF^*nN-1EHGLCm-&V55Wx{%cogkae*&ZE;Va? zoN&F#s8C5TkYxnY)Zct|n*na;``qVV;OWTiR$g-FWB#%q%cBHw@e|&rCg*&A{6Yh-sfi%}#}v zMMh7~ORX=`mx=usMQqln6N0o8b7fiBdC3AxV%cQ^L+|t1+}iZdco$M5tPRdgnL)AJ zyG+2t&k<`3k5Vw3Ix~eJEto+##MSCGeJT~QS~W9;T})fO#ov%eKTGe%ZpmP?!9`)s zrIZDfV@dixyTNWx59la5-3DH<_EdS0Ifu^e8o|;F*Pi0R`7uF>o?@qBqAs!0&2R%kj|N-``(ci3S+N{Ze3C~E1x9P&nlrVM znmudh`n~iQxT0*EVs+_Ty$&{A%)`}9wpK4^g{e@iP>)Au4Iw zI{R^Ki`=uvQTz&xiB_Dqr$jy}%l4Gin!j-v{7Nu$HLN04<&oq{KR)rPv7J6bX|SauE)8H^54^UE}#& zQ63hDrXM=?#PvAhpJ0zM1TtU4WCRo_B1>`aj=W06mSN^$$v$-^L#LR`{9w^3iD=pt zsTFrktS2gGGRi`E(Z=wttO9mf2Rl*mTHfuT^DUXbreRW3e;Q1nI$r9~<{UJfYRzi) zATYLqKrUv;*3s6!?|Mcf&hJiLw+4$$S8jJg`}V?gPqQ5xIDG>Xw_cFrT=QRna#`%e zrR*9TIB{W6%dAMWAAMs<&>&D1vv-eH&rpcrOO%a%cU5gVEa5~K_%(<^{I)xTo*4(9 z7R27H?y-IngjuWQXbGOG``Y6Eu|^TPwNY2$**54B+~Fy zCXp3EWyes;Ryz8>QRO3|3+fxqD@b*x8U_P~=z)lZh2+7>@lcB%Z!O3%*A>buvY+|X z?W^qMy%xyKF?Ja&FRTR5)qv62Ag^nL!>2l^_KrM#HWG}F4AujTeny+Ecc#j#y z?Z5r9;G4e?_TA+#9!)1Ya#@)s?0OyLQ6j&Bh<^iT0KyECjm+cK6;e97(!9*88@Jg* ztReH6;W^DZ-e`0 zml%CXO@7(=tT=J%Y7#_)tvs$kIsG}!dj2#BR5)<mhod?21vp@x}Etr-@&nkaXfq{DBQvg)dHDc4w4)-r3`wDVw|;RcifqtHdxe z!qrQv?R9DGQYh9+`DmTPi&efkquCvH1CRKUR9KCqdLO4HV}dSbJOW7g&L} zxJa^L^WDSrM9vb#6ihV+3$0j+(%fn3XWbynTHPnWJK8C&nN>R@7ev4!-QGl?4^}bB zo?Cs(x^2R+V<-p)dC~SmRw!?rK`-yxesWiGtUOM=w+i+5|H`;Qv+&#AZ zkWLR7(mxGb6u_v6&B=d1(^<-!7lf{SviE9H=czSwAF)9UOsCldYXgVR5#T3hW7OtH z`&2G*6+QFZGnBhbhw7h9t3R(bUrb9RP3g+%h)s1AN32WnTM)TQ$*BOFLbglnGUcp_ zH#-}qk<^Y^U-H>XAe>AiP}HZHjYy^e3g%-3&K_j!gs-vlmn z*XqQY+oa0Ll*<}Vq4NttCqH7^n9CGwe@YMEndW29krh-17usoDVxwY%ct z=$J=uz1VzkK9>KHchKbrJF}Bbk-gK_5_bNl(i)%GjGmh;DkbDg5$sP`MIQ9REN9Bo z%^R971G{pW0{u9paa>(^ey{yhnwmj7uWpcO#ko+g1+K^r^6Fye4g7;+A&D*IK;Mp+ zkcAqLhO=QzupU+8K_%%-fo0otv*R2=>e8fH+v;wY0muVghF#BY7bg2d?Zs9*j>bvq z=g(Fl&r@QkpOX=gU{^!=E)ky%2##|K-LcOcS@o_yLsEa@`>7Yq57=yoZMg>M6vqwwB?8+VzT0<;@j{|iFr@%P8m1J zC0i}QVzIMPlTW|7Ivs89ig{j_4se3`{xnZOSew8*R+89VOKi_sE)6$blg2L}@HqfmC>$`&+GutYoC+TSA zw%hsxKJ5H5VQ55z%!Mo#SiwHDl4Ir)Ivz87V(rMZA3|(7@jav60|Jn=`+LdQZ1XGn z3u+G2eC8c!OU*n=YvCt+gIhf0fNXm4lEIBh;_8{>Mv{^39#&RTLiq9mBdIvc#p(R? zhi!>geP+taX;>?4+^#{dZ$`6k`4woNTfX<`*(-Zw@=X#I0X`^7?Akzo`}H-7?WtNP zpaTMh=CjgoOR3h8|>FWSl)&5^3t7d}G(Y2L|4)IrypHZ8F=c!y)x@I>Ah$8OnN5#hg zU6pj;>!(4m1qGZrY(9pb@16|4y5%rAf>B%3RmWiJFwcF{C;Z^dY5?s)f?lf2sJ*4J6O`r@WDW-Ci_}%aF&7U~}X;yK=pf!_0akRQxK^O%ka~ za%iFsNaE42CFO7h^1)-70`DAStatfXXhfiSpl-iZeb9@C6j9J@+98zQUhP+nn!+LJ zwK?)0yW1fehGORT;^hsMgdi^>A=z=n$Mjr(1~2(db--xd4ZftscwKDj&4-Ok zxxuGSCin$1QXEv^)gUjvPp&Wo!fbeG=;KiD0zc-)^@fk7#dlA&6(;&pSKs&tzz~CS z#Q6f*ESomiUdjB=YQrrK29mg9m~nEZY%21X(b&o1Qp-0wXJFUv3R@J_-fN;xnn;(w zF*k3$i0{gk%3I>iW5^_5FbzH4bn1Rv9`9^cbyXcYOvTpE(rTLbTJ6XykN}^tWggwu z6N6#yZcI_cUXjIqfrXP(%Wa9gvh$BR-(9>9wEbi7_FAedym>;XEH#5`4yH7-<0bWN zBGBq%Jv}+yvNE`3g?#5q=Jn%H5oqzDB?lAP7ZX^43;~5YU2Qq#t%T$85FdnBZ%dyS zQ1#VJtAywHQ9gc!i&sLsHmy&t#l-|;v^)cYSaXO zLkH~LC7w`d39l|9O7V0xsN}k&ML@`y7JZ_-%*8;j!hwfmf4sl{sn28H$Y;;TN-nJT zJK>p@Ftlui96D@(bWj60mr&!9Or!%(IeiZl>0QfbPPl#Dbuqr&E?NQ?JK{QWl)vz3 z8<ihT8U*xIaGV%A@qXqGW<_paWYc6j7A*Y(^UQ zl6S6Kve2GK>1HWxTOtLdB!eryeeHlXa@s-7v#RgA*!)7sd)Uhokh-uSRC~8sr+MSH zWaYs)Xpqu+Ut4y2mI8`g#Byy1K^=2Y2Ad;^&mP}F^c99WSvgze)Gkq4@Odt(MWru} zzqIZg^L8~w#y2!BU-|U8ov>q%fzf-gCkk_~AGk^6yJaAi#erfF)?G2Yk+Vn6(aTVk zESEWs;y-2vkf2&~H3vn3O!J3l4IRu3c>0)nMYl0cJ(f;G1(V;#;vg+9w~)O?$?>sQ zI>WPWNx!k0nTAx;pX_7Mo5m_=v=z^uW%$7iX1LJ<60y%j_^u$-zSG=c+kjpk3w+`q z!&(MZ%|TqcK%u)CgrWXMl}5ivRVP(8KO23IDa?kIhVL&_(>2N_equB( zbfJUpZXY~jmq>2TjTsiUM% z>0tKC^YJP71UsgMsu^JVt%(dNv9D1B8w>BVZ!s>X_YmQlS(8r=GMP89?$Il>Bgsd{ zXwPBkF$XZ^T$Sl$z?bB=L>6a7`+7f#Gx z#IU-=n+Oo+#6e3;8HIT(wND!zGjxKLel~hn?z=6joxFKbb@1kc&YYOma@4lmye-Q1 z*9B>b`F7hV<{%4`t_Zi}yj{B; zJgZ47XYcsn%{&*~=&08B2rO=F7~qX1k{H34qQPV5Si~8xzN#6`&6uQ2&;|uCUAera zbQTZiFI-+GjM8sWU7gKWFp`b8G2hWMG!uI7Td9O55vW6T`gdu<@#F0!R-}cG-JR%? zKzDJzer5QLV8QK>x#r%$(W}H@qdFREwVc8dzlj}cKiW=Ud<~;w8@&gZq3Pd@$Wz(z zJ(Xi)Wr6o*Git=2Ot$gLLkNl?worcDhzCwfAtcN1!QI0OtQ^zz7EIs!3SGhoc;Sw{iL zd)w@|cq}k7ms$MA9IsDL*?fkTxh5iyr(_HKobD@8q01#XR2Nb17wiQCbOt z6vf+)_QhEF!$VQp;^A|Dq7IZrT|#}Riq*s~X_R5ZVe!rH(pSij_-em2Tlo0vldfU+ z6r&F4jr-qZ%I?AH%peAPvL4$dp&+issGh~C-i6lS_cYrT3rR2ym5We%%bE#;iB$w~ zDu``!9nG!fhYND)CGiIxy8YLvYqToO>|y`aM)Wt7+02xIk6g9aB^$3syHPLrOQ}wS z=b|VBF7|&h;K>Sq$1cq@Ha_xHqU8&%+i!YH!8ZQFtotP-8M{0k>C*9Y7sq-_wjZ?< zI(zsMpSfi4%dswae;gQd=wA4%AWhFCl$kMdAJL&PIobyMfI4ldf6Gi+&ve3KOOG;! z(=d}#JWH0E=W33@V5M=bK3t=|$BQxf((cAvwKqgW?oR4GtxRr}TW%QySnF%v*!ktn=mfriq@~TA4IK${`p0FO@kibXI?YER;HvS69M+19iD69lC`sLArw z5qR{n&l%#kF}V5~;cS|K^}dl&uaBarN^GxV!QON|3B*XF>vM)qQ!4t2xEL7a0rp9+ zfMb0)sV(3QPJUe;J%foI@~Jg*9yicE)j3_C;n4?@V@c=oyE2$6Rd!}5vjKW$XWckV zWn{oH=qs}{7QV?RQ^R*^2cT`^R&F7aAwn*(*NV62)!69+Kgh6UH z{qZfU=v2-8DO`_@Y2KR?mqm0Br}3FPPWZW{7ejZWgXzcGGRB|ErFpW3s)N00PdODb zvo$RuxPJOfe1(%Q848o5%x+Ye&lUM>Us!sT%EDm5NxO9LH0w@HeN+hMPxtJz(TSt-eVURq?Ux9`l>PU9xA-9c$>NFo zRq0NZEq@k$`C9vJkmQ4nplJh`Kuk(=zo$8%AYQbieGWF5*%@MO|8eR~KdQ+`kYU)o z5ED=<8`zK88!Q&roMDG>L}A$!smMX!`6mv0!RRm3a}=F<#t8LEYqQ~?M1p&#j$1@g z!6f@=bKFW&U$eECrLWbn$BSlyntO@ee5a_t=5`b+Z4n;}-e9$+d7AQ=%aOwv{1Yem zk}NK76M;Ha%HfKOoppKc!CuQU@M-4a77Voq?6~+{JY&Qub7@iE1abEwSgA5@QKFqT9tb}7Rc6=*0r)4 z6?eCS#t^Gfts3#H1d-z0h|{o3Z;rTMZ#xhdB;Eq+x*qZ%c)*{SF}g=e-jf1ToZ44H zSE!06bY9{s_B*n*lkDMFGtS7oVr*KSZ4#5Cgbr& zM04n>`ZU1L-QYRnQ7P7c-y zgTd`Y34%dnN@$EGY2r{TVpDl@r>HVQGs3G^GjW2xyYu~umqHMHrAc00uZQNJEYHONeL+Cz{LI34O z^Q(coD#k+K?QHpPy#YBb5#39?XGb@~hllUiEo@rJ^dB4azckzJH8xmP;Ho5kf@X;6 zXgqUC*{N=@ESOo{LTN_F$O*K$wv1;_S4;Vz8<$0!`&=nU_zUAIQB2j(9(ha;?}iS$ zVQgA3ij1*kO-tgK)sLy3l!S&sVk~wl0Xs-M^=#a0*Gl%_s8cYNJ^N)l-NsR^b1>~h z%~)rTn@-?t__S6)Oz)rN^s(ZbGi!OiL8h0hH6flUOCFzBp2E(?K#B_JKCa&Fts^f) z1AW;HyUET6e!48jBg{d~>`D5YE^Su8#g9uJW2eKo5*;Xwn|Yg!tL~;X&sv8|G~IqV zeYrKuND!)3eLc4v)Kipxgz||GvR~4*A}&&%fiNASl>yW zwzxus$ETc}cmlcSrL?tJ9b?DK^}_BUtvg1tq- z9QaTIMqp4c%t6^^vfQ4m=KX?18ZEOv{Sh63IlMf0jG&4zDvI>bU-dU<;MB@J*&7(n zD--wwYK{>tM3ktEa@rS(1F0gxgXe+~-a^o^Q~^ZlIdEc$*C*|RGF5X7{c>~^0T#}3 za?$pf7dLDRmLG91?^WrFTt=N{p%z>;B2i{@{(pkS6K`v5inc5YzJwK-R8?ID0NA?C z4kE7OtJ|aaqS=hCo=J0@{Z99Hc@0F7y4m~+;<<0TCyljvNN=tejyp$Kw3-@-Hxu~! z8xZWXhAx>BO?jo{rLz|4FwMq0=dLMT;vwyB%P=+F?f|hAbche)JR-ow=JwdWMtasG zk`3eT4wy-VrMCl)D213>6$+m7HLRnSf>t4g^nE9YH?HapIMkUnai@FO@#u)`bc8GH z!JPa^vA`*Ro-Fw$qw!j@?LY|qn?RLVGw&Q5jb;bG zm$PSKPDLB&()40dV_rI@`BabD&N0Jm!shPw!M>g7sJxJR8V*(__cfs(S;-?x4iip~ zO2CA&Rso1rC5}M+XGNwMbxvAO#Kcl)Z<1H>zCprcrXnM3J=7-@O!fZV?n&vVT zc2u>WV6TR|J}v5cV9o{jdlw361W$ zQoYyNc;wr($Cn6s;%(U`$nv!9+PKEB-`kc5KaISFD~{yHb9WPZkLBgoB2T(kF9qKi zihLK*Gao#Cgm9Fd-I+fZ&1m|vAZI(8O?+40VeT7fUi)3YVR@UWPtXj&UvzuOL@S9= zd8aD6idUmM)&_zq)tggI@9c?VW*r{g3<(-^HN!wA6SXs3frbY7fughk3rhs4K`|EZVq6fTqawmN=9U6xhZme04`sxbG zGe-ZdnAgOFPzGHPEKi}xtEV?st~tu6mJ3dWfki$}oyYb~x%jxup`3!Jriy;}mAvdH zjBSF8!e0(+1qa{n9vz+0_q&xZ`~W0^#w_Q>yb*EklyH=wqDk1kPt!mzG)&#sILZ>Kz}gd&guC zhsohUU-ud!MGdV~8(u(bo(`e`@3Y)bbD3-W9{=4p!WF#B|L{Wr&4S{7@yu{F#%`c1 zf4SrGB>yvKogm;6l;lS1|MEhF|K0vyjvoudf6=%qiFbB@NSR@dnNW{l5s@tdXFAR!hRwDHFB#<|Im*6mry1ov zln9Rclx@@02uG%jyWmW1B}J@6-AlFSpp^U0{0A(2jy3gBknLyY#H$coo` zVHjFho7*hX(UZ#<^x5a@>!3(ADrh+Gl-W0ZMK16nPeOi5o2 z?RmRfR^G3k$-I9QK2(4{xkh8~7w)166NM=5x}y3#J_iAB-7;bCX~1Qf`#3h!Y_Cn6 zft)pd<+Hb?FjU|4VN}{(ZEo5Jx6AsJErkmD^s+_;u=U!68h_Sn30y$v)SznVcH=Po z#R%Tn`_UC3@-5x>Co$bV?TZ`DyK6cEFaKD4&o(~umzb0HtNBH%4hE(c>*A?y5s+p! zz2d}nai;p6MGX@-Lu7|*39qb7Lg^>IB%7?GiO$;=#@IGVx3X0Q3hn+DBhN2<<|k!P z_lhKA8@u|Q)C3gl(p619PSXh&KL`U^FfC(qOWsvN71x-XH`FhC9^@|^?#~0sghfn8 zy?LLE7C({x)(A178Qfb4+{g)(rW@&UJ_K>9)5QV(0(`!jiw5(MuJ1egIw|=zNauJo z6RoWjVj7vAy+yjc?a=H@GUB`lurM^xf# z|9ykaTrJ4$hxxAgX|DeJYxC4=h;o%5_W@L^GJC^{HaqI%sgsN!VG@}i&y_qlZv(!Q z)EBu4W1xfL+$y|ryt7)=iKvp9*}RTN72%wnbC+#LVn(9mv+p|)1~R03tru#O(a!1% zzB$^r&v9Q=ImXqFqFEryWUO4R)pKvtw2r06z2$sKDeMc>3C-H2Nx}hx9FgfQ3Rw2s z_nBes)A`EW7hTrL@@7fgMO6pQxy*>;thglUWSd&*`E&8{S@`U0_*T@{{eHJ)WJ)h5 zC-4_O+`F0k+V&@0j?IwAg~#UV$QIw+)m!UxJYKfAnBaeSGU60%qnV#YZD+hgf_&GI zr@BC)N>?;DT@F;ZZufja@MJ~g%JumbpChLkAb9$^f`yDlycfyIili4db`=8LqCV?Q zXj>u&GUH{l$M@LYk;+7k$MW^zdK@wKWOFcBwj(<`m7Bwc+^|kYKf(DtD?!d>h*25& z<~E5kd#64_PpXi&+c1l8upy`oJZ`YQIA067SmAAj*f^SawWxCc8o&L8&Wc=+&PUHP z*vQwsvT<$$N|CZF++IKv*UKV#^GNt3mkV_O5KRXoY^))WJr`RhDp5zh)OC4CjrBTjgQR5>Y@AZF zwhDYfYlgN+R0mD>>SmC77;jB&(*27HowMapVideFd40YaO?$ZZB9zDH<4})PAL8sO z4Ib0`nfoA!jX*p1{f2tDfvxsZ+t-u)r_A*Hhez9*ouL%B1b1}v!GhF5oU%iU1<8_H{s?m*W!>^L2`xP04@vZ?T( zHdClYr1yog{g{wVMaX*3&#SNTw##M!VZDNlfdHUyLi&q(@MNJ@0Sm(b%!>OF3tUOC zt7zxV^Z5r6z#aLw!*?#943I!h49kMrDuGJE#sQS%SwStFRlT-KjOnli_|8?x&_awj z>znps^n}XEauAnZsDRNxoY7fcb6+3r>@`SgriWZkZnppPJHbF~PpIb>X|n8=l0o)| z%RJ?R+E&PIoNWqb$SPpUxA+pr**gS5Af-t-;s#Q$Vqg!n2)Vn_Sh7HleE4~KmQ+JE ze<46YE;)-8Z`5ftQys09+;9-c3Visjyc&zmb#rsOJwM~i2a(LQy?x8#%sz*Hd`&>m zr*n1HSl2`kUo&B2`AM~{ai;gu+$M@81fc`f@7s-p_3wY?>;DtM7Ypot1wKG9pwj$> zbKZc{d|)y{#^R>GZEZxh+xFM({2aHC*t`so%=Pj^5c#UeW5SN-_(C6=Act=%SJ0?C zCAPXp$XFkGN;n`0eGR=AaTcuGNl1je@jFc-Jef4B(f3H}QF5_DHU|H>&*YE!*5^Xd zc`5Yf-WQD3Vp2MM~B$^cI#&YB$fdtJ{qE$l*R*5&^1FQ~{qn2G!0)>a-JuR-t zAw;%YB3+;UHp(~ymO<8YhV7-_Aqdf63ZXxU9*ySIOhQi|+mU zmS_zqm(DBAsj*zCeEsB0^C&0-)S!|g1hpu78}-*PtepLJ&}j9a)jBMYq5{9S2?SgK zU3>F#YxjG&(+35XB zz>UL$KI8CdaMA*VK*<29ive36DYZWDhDg_JG{~7BJu4GFT_>n2d+$10We7VjjeB^b zJoB4Gy(F&ish&XLE$I@Y?iPzrH+8h)bTn8x%HoQ4;87Lk3VX!H?d3>}T^z(v`p=-D z>Nege-r4WJuBkfQbc_$gZSut>Ky4?5^Ufy4L7wX_Ok)K${r-K+N6v102JA|kJV@!+ z(Qi5*i?EB(#bdZ)qQCtwiB$lZpZKj12+zfA6;*Ci?|~jbleyiH-~yw95b%!d9^lW{ zi65f20qMY1gQ*9(b=(BrDb)S7k6S?rT*0~T`D2=RaHqH_68~=Fp&AHwZ8ZjS;_g|Rjh_9_ro}~FKHcW#)DCox$RgD$T?6y3U?fI9x-Vj8X3QT3A3@>CDr!iJ zU#F^YZuTvx29wv>m3RZ`)Rlv$P-e!bhhT?*gn(bxV{kiP5}+Zs|1hA~CP8%}I~H;W zQwC?)4XXi1<#TINQfB5If>g6z>U=6wMY$O$KUZ(?6Q6kitDWUeIeXsvPCP_Z7qVx$ zCxSM3s&%z`jOgfL-jR4GvG+6wJEt;9*m3DvZ?v(&p0JJgRwk|$ z^>bgjsQd6_@n__LW+a~;v)-F2@z(yqTOCD>s67#xv(v@n*YXmo=I#0_KAm-=!&h!5 z8MdqTsZ`IlHGRHdCf1uWX-Pip-S%Oe~N$<|VmWS{6f)k7f zD*N_88~Bn;^!e`XQv=K?i`L8vcW<6Md5Jst`fLA9!|NYo$`XbeN&CJWzZ&DN56%aX z<%&Z?V?G%p8|gK+jRawJ{=!0tJd+p?X8$Q@0ygS_Tlpv`I6ZCZ?}q#2%9ym@fVhgCiZ5gL#zxm ztVcDkwFf^s3FKioG{!1bgL?4Z6Q22F8bhN3#^f`OJU+79jgWPIC)V`n>5KJJbA4GLXsko%55m(B?oUj|7Fowbd zuYmn~A;`>05v>F&W@PhO$dgvsr@^pfmwz)8#3itxKj6FR+B2-NhMIb=Slt1LQhg7k z!;fZ=vw22Jfvb82f2iy75`P}j$2ZrG$wyuOAcTK+|-*%hG7OCsw4RBOyc*dd5-H^9Om4>p2y z8_f>!TwQW^dG0ij_(%{+WXn7QWZB=4iylc2f8tor1%8EQ4L%a5+-xN-F>t>wR-YIT zdGVG@Pq@Z90I>jkO+)$m53Zxv{}IId7r1g@g-NH(tgXG=dGSOu>tOPMUv>ZgyIA}$ zdiFnSym0PZV3183xHiY;BO#fF&WQlvZAlHIY3iZT73c-z7iMfw+5RiDa8b{d#Fdiq zGfXCszX!%_y1(vT_=my)D3|8{A_@+a0!JZ~Au{LCa_KozQRt(>d@tqSb$Wgn=T-NI*H?j~Il7qaJf=rzy0aa5-C_Lg4CG1GvD(kY3GJ6M9IlFz=>w^>PK zSdl*)BI6gP3_^q)(j22Q@CDJ?`1^-Ud3=nAheimOQWa`X7I;56X5)SX8S?P`$v&on zDg*+5I^K~4Ihp{OQ)`6j4;Fgx2-7Y&-aK3hSWN$5tuE?-SMVmgviRPrQ?}30PH;@Y zZW)=Z>b7Gw=T5_fpaCQI&XwSD>A@2b*!kUsqMO4U|D)u@Uk!{g;5f%M%o+a7gqTz% zVXL%E!X{uowmt{aplc_(2Q{lfr~B|>yt7vUO+1ZOyaTub=HXcy_6#e`BdBbg%a)bBm|Dv_|5!jiOhVHn~I{stA+|YJi(8r zLpgH$R~$-s^A5^oDTim>!N8OJVz0BFGnP#nfSXF$KLf5SvNls#3Wm&v)fxvG~@XdHHJDeWFxg8@-WxtPIJ#4K`0LXD-Blw#WD|A+TU8?#=L&+?ElcL0aWRBQ)C^5 zWC8ObExiix6dyfPN=!4^G1A5v+{f+C-=qepSoG~r|Iiqik1?MyGdpWYjTVo^H{~|_ zy=^=7>)5$98zHN*tG{GdSrx%dH3F0|F17E{Yd>+k_X1b}<>!|E%T5EloK{801<>Db zA?g$b?P78y=A{%cyjws?ySOE#fs*D~8w8+JV z+6};R24#&(o*~A7mBP5Os9+B)7(h#?e>8K8f`wl^aXmzpyn7wOjk%U9z_t(@^01Ob znPW=)baBtA_A1se6Jxy#O&0 z0nXM=!h(YH#sku6q~ZW!ZE&7r@8qX~1OFtg{{(`Xy8*j$wO(hz0I@Zp67}HZucBKx zcLO=sic_PCZFc!)0z(8E507BO(G}mvf?5cdfOn6H=da9yPQR9rq`dt!;* zZ<|9M2n$R*WyVEh-@8sEKhb(X!oNDh#ov2zmjX0nlviV4Stx~|#e~T+O;m%sy^L{- zAS0QNYXNW8_7k6sw=Gm)T;CChZ^Y6`M8f2v35=>Bmyv9W2rjfzpz)uNrOi|wj% zmZErF*0;bv;9N-x7w!SOkOd*s6WEvcW#XkcU*Z+U>r_zXc>%k^UlkpXvbkdKnj0h^ ztYsLz2MEqdHu7GNvzS!5m6RpP8~7I1%3o6n2}sl^2jNL2V{DCp{~f9QzbFA$ZQ=7F zc`8qjKb3@(E^jnZlD89Elghi%QW_FprJwV0y7!+zH7K?I1X;`iUlbF6Yvmhn9wODD zgugIlhO~ka8-G0J>X9jMY%$;wudJZ1YY{1|mAeEjbK3GeO2??k)MB}MW~&80?& z@0R{(yKnt&bulklaiI?6Q-!l<=KFc`lDalAXx|3-l4PF*SqR~e(HH7}0L``r*tN;T z;Zduz#nhonkbsd?qQk_J?}K0Up{rvh$XF{@-u$8k@o09GmX0Z@Whpb`k7E8pmwiVD zg`oaLZ!3b=jY$JD9^s1j{chLhQVMw6zrgtwfEe1pJ}vO;yl3YwF9w7klaIG8(pAKl zKIQ8-)zTdeV8HG-v`zLiE^7K8=hQvj)e|L)%k#Gn>6_qn31z|0{NF_Te)^Zn?S)Pl zx~mG85&I2x(#Q1Zw_}I`$ytRn(g9b_X~i!KL7ha!AWcVPG1da)oNRoeb3%~lU_5a0 z_zRS$ugt~7$^yT*@ZMEHr#PKz8P~yIm}lPoBl=lzwl<6rXnQY>B#VpY`i_4()CNsu zQ@xAcTz_$=HZQ3SpZLc7X=Zeaf#ZceKX95k4UIp4QF?NjnMcD%zg0^dOt^lyVbV;; zfNWHdYk~B?7vP-g!YOY)4{r219z(y^Chzz^1bB4K?7;JAAUzi9obyn2(%2Pe`x`ww zu9&_lVqrSF$fs50eA6Bv75aA&FAVtvaUHUw3-K4sF9D9t-#Nnn@7<)EWpE?@lsq+c zFaH}FRWa@>--%edevr_wg7Fs$&o)}_$SF}yzbQWy;VW+9=ophz`^AWAb3Ukr{Gob*HCbd*y2WjYMkJf|I63B{RXlQpeu&~T2g)Tqvw0sb(&E1OI_!= z(W)6K+&t>$j&ZZ&z`?M8;ZWegEgf9k1D$8B?3gg-Cd?$utRGa%3I$YF!n7Y}Q&Qj> zo?~{zO>#d*k`>4Wbnrq~qcPF_(gxgCfbVs1Rpie=QC01`0|}}fMhVz^zYJ4*0(FEO4|@YTFINBJAc(2DL_bvz9b`43k3Ht6W58;)0g)F zE{|86e=lT$3@Ii9rgdbo(d=5W8W@|`EXY0e2e7uSzu665MJSMSqXgFIHjk<1VbazP z0d7nZB*6_t1CfA!tJ!X81_N?s8sKE>Z1E?{Ruai54Tm_)GWd`2?Ok8kpYel!tH|!Q zHHOV}?z8vHNgrw-s^gdQ+rZZ7>#g{ppK%sF<=J&XtWTU=v%ZqR<&x@?Zm zUheOu{3F28KURhikqucAz^(h${x%!{dH?P8|B&GQw_y?7OdV%>b1#BDi{8^c0rv9m z61$?Wj3AG)Di*UhfxH32yB=LB3@89G@C2~lABXDzoLATc438UyLH2Tw+eU5yt`xbP zn;WRf69Y2J?13{`ph&I{i`!`^bsobC>*cIq*}jkMtAdym5@Vut1L)GvcSf4K!0*zT zjVK__G7?0@e%+z~YWVr|zpGJ`-M3)IS>SkIxSBA^5a&eGHC5+8xSd>^fg4UD)228@ zYm4G+zKefMcU2&y*?)qN!1>U>=AjQb-L~@-ch_G5B<`;<0`}wMXF$H+1_j=~heu`1 zSL6b}dJ_HY94Bf~Om)tROVmA%ui78)dV#ZYt0|_$=+Q4v{Ui~68<%b0d)nbhK z06UkEIOnJNMnR`R4AMCL@xJ14V^6J;--=3jE!z`BAV@4EwIUqaW*r;+7AnIJP{wcZT+hCNDp=Y^3`U9_LZIbsMn>ahg z9rB;O=e8ki;Ee|-TKCI7FhRCbq~g@yU+xWg7F2hQN4UwA^g?rUN5BL{Ix%;(>sn3c z8jPd#)PX*I*>;xT{O)`o3YOL&_~&LR+)xi>$H?`~twPq5J=U3?TnB>PjlWlYuOM{r zs&XT>!(F{{V(ig@7v^Q1r`qJ2CzhJyJeDXg%0F1eY=3QUyiavb4seMF8fy4o61EtFSw4YP zB8)U{KoS9`^Y7!2x_?yEZD}wTi1C#~^U6kvX8^yaaphL0GG>o^ElaOPvc+2smHy${ z;9F1+kyKx$ngu#NMeL-mrdyn#oAa4thAt*`>y3xBFOZW);>$Q1rPwQ(-Kup zg3uL}0;VSxsmWS42E?ho!iFV5xs$l_qCK2cF{*?J-FfjzJK&ht2OS8;knnmybZ0K0l9peeKYE9ZQ5OK>He11gB}^IH zAKUsbJ+;2R<-6Q+91JJjTUdc|Z~k}(Q8sYQte1)30N9pJeva38CcO9cS_;!CMY)3f zQzJm+;$@80jyio7iOK(#XhUd7?;}fQTO=Lj9Dv%Io2hakJMx3sp)H5)>VI5T7z&$A zUm2Fhh3m@Uz7YJoJ2amSz5|_Z%kp8?Uge)E*ECQk5fcq}xc41x*OAMv5Tu~?ctqzC zSc7rsiWi!fmg+JAX;qrX@~Sd>f$+mW$%bTvw7k%*oIkE=#RJDR!XIt{5*$WeQ21nl z+yQ;H&2~4PdefcPhI|@oNM07!(+_3hZ%bKEnso=2TTOU|mS02rc68|k`-qSZ@g))0 z6F8nD`bRVtg&+)Y0IJEU0l~K$m-SP9Z3Q|Y(JBNjRJ&~N4%KQ$fKQMaz?W`1fqK<0aMejFIJ z0!$G*{CJs!V58aA%=bK?;h(TDLbHM2V&!9({whGPLF`4`1D#g}I=c=5xj%I+u(5vV z+JDu9zu>|2Cp-YU>Td}A2@n4<`u#GGg>yk;%9;f~Pl<2YC+gc->9skGmV#weuMMH$ zH(9;SocC+3+AE9KJsO7n+=7&cUjy{xx5;x+GlQBdw?2;CcD#SAk{2Iu(dV)6fHLcF z1eY`XHgLV>7ZCfwO0Pc{zI3WVJ5lkh8$1l~OF^l>c56ECfbB)4Z7a_6OEQx?j|0A} z-`6p##O>jhfiwN$G?*57Z+m<&LKMh4!u`H_@ya;hn01qFmAKhLaE2H=O8lmHRpKb; z0u?{tjk>uI42dM1MV%Qg5pPnyJ{ojlN*yu6ewCwHoxc<-O|ubZv=k0*Ugt|A=I!*z{8QT(ywf2{L!3PQz8Dk{ z(m`w3qiZYXx{H{(NL&l*={R~XVPJ@R*O%R0l;#AWezJ5*YrfdI2l-g28qj`_TpYG@=mAo?)2sr7Tn?$56-V;yX?0@GF z@$lWrSnq+)StsAttCbUJ+IXN9Rv=joFVS*yTS zU@vq$@w(Z3(h&me{@)m;lC?i2S4UXLmpz8-paVZ<6{pr0Z`W; zG!0N4dcc~8^4bG=qB`+A;D@fK73V#tY~eHTKdNbkoc4b@xh!Zh$6TUOc=Ix9XOi7b zk&Q4{fm}i51%+To9#W$1Nl3H=!Q&CNciJi;G6nmJv&r0GODOBB%rkAk0>lA|?+5YR z#0SW6hr8qjdyjan5nTpCeVTMD@oa#mk432yk8pTt<-uUDQ9%<=>v4w1KG(yU38EFv z^4VUPYEVQ&v8PlXqNL~rVNGV=xfHD?DrHvJojyYwV7F?>ZB*7cZ0 zf_6iC;N5gSwavL0O)N~`nxaFanKnrZ9;~EGjqoYP< z@1-tJiY^65-f0mu7|sWhH~-E^MM%qZg6{mdP5H+93zAXYSL+?I|9sMv9A&x`bKpc% zFR+|3U&zH(lZ+qGU0k7mB4FVD{oRR7S$W(KyE5Ko9RRtbSBfrpVRvHq$uL!tl>5PpKO;hbR0J$9}GOx&av|V?{ zohDJ|8)cp^ApSOm&wh|b9VNfSVOO-3V)sdtM1bqjR2)13R{oQUuM`lkfd-UE0dX3j zE$$M4zJT^i0&r67z%W$!Aw7T5r@LURJ%}6e%Ovm;;8y$s=o!nrW>4&Q&iAs!lGB|q zb7^suRK~R-_P@vO_Cx`;{}b~t<}pwgg-gQoGq`V^KPFxWv5;fWVQ8_KCGc^1;_%-x7h^F57S zaCb2zUm9?hnAPrY5QW2=lff!D3k(FH_jnkudaT!|b}n%!1X`e0*SKmQAgOB)<2q8E z0EAB?fcrTonjx-pK2`xe?JU@4@)8o~kZCn_lV0F&wLWS^z(J35e3EJRq_|)qR8D=!nPf`p7ROA_*?KtL zFg5M|)VdISvM+S!e<5fQyVgw2{Z7IyJkyfM=44PZ5Q_-OUI=8( z4G}-&X@e|+og~>}yVP^0+G zoba9UkXvso7Q@WX*%~|i9+i3!6z|8xYc`yInyhp9Av1?y)8ZFP@W@+XXL-))eo{;R zMDqcrUqzKY=7k0cq3DTPc<#+dt|aEm={x`Q$%)Uu(cRu|-v$>Kf0oNNI&nfPZf~}H z7{)nKFFoIxPntBGnrgW?p6ed-Tc=%7SG@cDofX<{rpc}?uYUwv01{VzOoooND%EKS zzt=ro`D-~OiMgQBJOjI<0BJCQ&WF`J`xa|s?{r;p)u}&lnH%?X++9)VlDFfL#|82i zMkwy-H@iKLR4B=b*{#+zcf`3^pno+#e;-b)L>OfKzmm)Ws08tpnK3K?v}XMKaovW+ zw5ZfH82IVuUe!6PEZ2q**ajBfv-3k^qN^rrucGI>1*|pJD*jqRKk5qk&Nyb+E{Oen zMSy3YUwx5xhERQ=R8HUZCqdc%lw&iDs6Q*Dl%qL+wCL4Rs;^w)R0G0D&S~A&CKgu; z0eNn4FHI(Qq(!Xzpmh30uWaNaEPaJ`hHl#d_4SYj6ckkZoie=lS|d)=-(=qvuxj8@ zDZ8oL;I|Y9e(G|&^}>%OGSO^E*QozNw&XCVpz1<+wE>=c;YPa@j|(>)t*b+H2F`Q= zolvOjk0#fjsY)|w+DEiWWw0aa6$roZvsQkxh1Iep{b27OJWmYY4pz;skUDwnI52PV(up;vuAFE?KI(w)MP^m+*T(7~Lz?2^H7*D)(7($&0C zaPPFv2@T?h{ZB<<-qp!EK~EsZ->}N!#K&cb8vj-P`%2IgM5O?9)#dHDBjo^jq%Y_B zG!wYzw@+RW zbXd)#B48Q1NRL0zqy=ZhqcW=trSLCk(jpxxX@D-Ju7f5$6+9zH7`;2Y%A|EypnWgb zkuual`~NvLz`y@rYJ&d5PE;0gQ-9|@NW*8)z8BAhY+TjLSk&T9YB!t~Lk84Z3f{RYw|WvYV8B)7f%!Wv5$ zp`+a$6!q4kq=I^&+Sea;aB8tkja!C0PVr6bu2c8uMrXwhL{}6iU_XU|BBD`m?gIu% zfUUh*BS_b~lA%~om9DkT3t{Rv|H*`Xu7}-*V~m=ybNAqRKA^uGN4CfWCamqiq=pnn z3*4BSTf;$N&(`f4(;T|EZ-$bnrm#ex;V`Q2tIn_q)z}!fe0VEiLo{LEyRz7V>OVNv zsqjvNfyLe}_yO=+vqc_RT*+|M$6l9@TrwqnWmp_tmS>TUs3(Fs*gv(isitqOQY{*8 zITrYJC~pt^NmYc=O#{oEaRN+d!F6Mo{5^6>Tld|6tv6ICRxYphsW z?bNO{^s#@HUm5|bebO`>S3QSkhYvo2MLIl(@OoAbyJzxv{$P6d^zd9a5Y35n-wujt z5A-SNLIzDTeN`CVB6(Q$!bUO#-3gW#C=VMjtkyxE&U7?ANUsnGr&cTsj~$?>2qoR} zoKs7b8yBp{#QKz}z)mI2r+1H+>S@PZKMhovH$I^jnpP)P+R#l=-K}N8>v4!zpW)3*gcr8U|P(!(X@wCI`pg$B(&>;lpo9;dG{Mh8Q`i&>jgOc|MXm3B z8!hu^L+|vohkrDRU`NzHn((z>kZ)qG7DVSi)>T(xQmXJ4;2zP@HE&t0w6%nm8*+7TFH!_e8AZ@+O2H?e`3r7RaAQI>KWBb45#G{LQ)^W8dL|W zS`z~1E08tU5$m$p0`eU7CGWud7P>t?X6BkDanZ%3Yy8XfpdbAp&B8*X^6Jpl4rS?* z{S^so5?_=Zpd$;OB;I&6nP1GrhqN);-Ehj|Y4O>KNlEHjk+{2oO^DX9R*W~kAOz@| zog`#nUZ9U~E#_ROc^%fu_uLgChFFB7S^V~I`bA4wTC_0{JANaH)CWVk`iV*qjiT9w z^nUe$66wB^6BeQ~o}sApE%ROy?6&@%3pab91kzLAj1g!d%~w!kMxS;+gU|A*GQSav zQOHzg(a@y2MSiSXUIrnP#ANHSl@bn1s&VgViH{kns9M{IrLsABUMICED+VZXU?1T> zR7#;dV9!#T8xiwP9~)7^IOq3D8hOFnLOocGl?yDSPIlGW4@&y;tNy6_<)K6^8-CO) z@{$s2e&*Bce4|}2>=O_D+AKoJV(j*D2_;`V@K~?bY(0`u;MWxlV2+{1Lmvpj_fUg` zbMEcQC+f^9JEwRkcCyZdum4i}lCmYuADg*2cgAX}=~{Wphr=h`)HvUuYq@=}U2VKK ztu%{7w6%0HcE+2tNrFBGy2DgxiaT7-`~geh%ahWmL|0#?%6-lzB4WKBwY|eY3~e9X zRKafl2G%^)vRaL0xHvZH2-6dh?{T-`Wzgx*-5EFCS@X{Yid5UL%EW!rwNYKwK4-1l zg+2LBbpVNFG+_!bb6kJ_Z8Mgg>^!Va6RqzccjqNbGU$qyWq*3R2Rxd%y?_9L2>`-3~z#(LNEEk@6k zveidf{D(N(w`A81UGv)pd;G#?P=#VPV-u(QnxhXqq*)-CZVc~lLp%CNt(=#8cuFBt z%|&R|V^9Rk`w|hR#tQMnv)Qe^*W#2U(`J2ack{%HZ%d~Z?{Dd9dd|~W%S8XVhO;{a zf!O&Q_@tr)0Turg&W^EMJx}}CM9YWAOt=(WxTmbqb_SQt5~OE&HP*FEUZE9)bZo!J z@yYuAxAO{-^1~IX)J~dKd{W*VrywD#ceRzhvhXHRSh$B@S!uOo<(2iOG;6HAJDu z68!84r-<}4EMo%G97hUvYmwXHzcf{EhNRl~m>LAC!x-eaEPeHeq@AQb$v|S#Xp&x7 zr;RHYHMG1g+nyUXm{0vypLqB+c`}_SR%yaiw2XtEE{*)O-5b1^52o)%emSC-3CGJn%D-7X%V zKTIQ^kzC7y7^9(rB8QysTw2yHaR^8n7Pojbuah=J2Eo)mCgL>bi)%NOt+f zh*q>dB1OgeDtQaX5t*tUIpLej>6H&8WOPX6OXmxQ)Qr_(f#HP5G_y(Ko(qC8D~@|J z>;@{_KZ7K7!1nk~qThV<^3VKz9SYdVN@E3#b;^MLBu{VDFMyCqlng%Pn4H9VbQhxg zxc;S-GL|y$ROeL*<>Em9A<2Asiv1l`N|TN$8EtJyi~7CdWFzVQ?i;|^(*S&j+7TMQG&|Wk30nqVKhsJQSy%!hC|aPz$CzN&FhuT&3O^9v5e5OsMdfBs;$G4`l?tS#}?)7AhW5E1gud;szYv zu>+Vqch!Pn6;&5f0LB<*S)8giiiq=0<89=y5v_qA*%^<5;+!Xrda3MlV~8x>XP`k_ zoY6MH!k{*DC2IFR{=w#WuT0+KtCq1|+O*>uIRDVPnO@h|HRroHv=d9_CzSF^VQ~WA zY)d|CMpSn+(rH4R5ji?%gsMVBt8M8)Osw?x0A&R|!3-a~srP5!!@Y!yRs9G-OIJpyPkLU}-clt-Ygt3F|+ou%C8n=c&Pmx>P}(6OsG z)>Goed*;LXBL$Xs0>Wl6BwiX#Uf(Cc0_Mr zRH>}#TcgM4$Ufh$Sh;8CtF~5w6^j@#?kb2&GLi&DbVZ#_`VF&P$+f*C6eEbbNVZpU zAdFw5u?ZyCsRGgKJu1InM7$Kkk#6p(W*$(>LIMT45g>W<6-xQj_mf&81J#g%rU>(4&DNm7}P2^Y`~%4@bGyvSoRh((k$qC^GRJM zrBwm{@VZ)KiH{h{wr~Z`)x%h{jct}`Xc$LcGNI0^waMb{Jj{+heqB`C@#4Ih9*+S% zhV5uUeH49!e!;(P>oBOMy}sUjZMt;>`xM!|UMHb6D3sP^K4W?I%`Aq%j&zl*Z>4N` zk%*v?QK49qJ2Wq`5UxubBwz}?s=ps|Q|LYoi|e}*0+pC(w@hES(0tsQ-Kbw}{m!71 zXZewvtRGmrZj^?O(B-Sfg}Q({4)QZ22NhT|&fmh=cLAz)QW9>2jV?g*7S@U6;~q2F zhPT>5mGxt^)L7wHgR%@4APRbSnnM&Onskz2Y;qoL#sV?p^2 zt=02wV2QLCn2(TgtuI6#+sb;t89LI-9TTDU#XfPl+e{l8metn@*o3$FYzEY79(z?3 zuFw`ycF&}~-mYtI->CQPn;Z%uQFs763&nsuGpcwUPxd%t<@cxuLa@vy2~;g-%fCFJ zs22ebP9-}R9+%n1P2Lf+1U%7%MVzj6l&@N0PYc%SFD(LgeAyBzjCv71-J1-mbPBQl0DASiW?uCz4ZcL*@)_f#TDW4 ziG+UIu|?OSsl%^(dgnyvpk*Iut7{?U{NJ{kCinJS5y$c{8P^WCPEz6GL3!%saT<^{ z#AHdiPoU)w-{g+91Vzr;HKu4n4mIz?$?}*ZtSB{V>e{*RGS;}pha=nQL$lP|$3%GL z#+k6FvA9WNIy8*ns$_6-P>ENhd>)CISMvO#@(1&+g!!~jGJP}WIll|VqyDN=o;MLN zs>Mb-!ma5*<2%?qhjuCxKWIzR{lX0*@w(WlVemp;6AiUE=(!@JlloL#+^(k7uP&x&1aa? zV!RBCA!n$1ici~vXE4dFKo0h3{XC~r0Nv0!({&HsPDr)pSLoO4 zlKbWDDL|}2Iwe4bX@A4V__F;CIad$kpCKpW^DoEFHH{YGzxgTdvCW^Foa3EpwDkzT zJG|3xo~pu$c5-mtMau}mE)ZJAFTN;v|07TB9Jq+#A-h;YvL<3f{rj#%_ogyxb^Aiu z1D?f4*HGk1>fW{|YyV9f35?hD+uK+Zpl`#>$?$jGU9qasX_d=?pkznWY1LL#>~J=m zQFr&Jbg=;yj58srl(KHg{UyKDGLI$L!(uri#x8YGRe66Si-58$ePq86Kgfz<-5099 zFeojAgn*0D0Mce2d0^0W%kwYxUv|0dNhZI*m4kw(=z1@xhui8c!klb~94k(aBhR^G zWq%yI_a^=IVbt;AROY(-r#BRhGrR3}RVKjkmINKCh|WD5pgYtBjvlsvP=q&1%B0K4BbP zzWc9Fj*4~2kIVac@T#xr!tbkG!Z*H5%fmXfF|^m02`UKXgq?ZUHkyhqe4<**Y|7(m z=d^j#3;E*f?bXQ3jloCNem~@)$r9FRXY>c}Q^}uj-%M>v?9p~>*T`DaUhRf+7_H9w zingsEDuf*8q*F*ne{~yImG5tdGqG_ra=8U9U3=wpqB^y?MYdpf6a6W9n(r`w_jSX8 zQS}&kExEW%$InndWwE-OLu)sO6;O`6JSI+~s!~oZHHnoI;b(q+fqCBf(h9*0LDjYT zo4TvHLGi8QcKF&wbQjP)efATb{XTD1ngHXz^>Uqrh&c#LU;&Oa%*3gk651UuZ<5oi9jAim&`@`7{aX%WeY_MYBCf-zmQ;p&M_ zLCv>(ipsy9acjDxqoe7!^8YDV8_()f3}q&2+ycgbJT5>8?Fc zB}GnWbl*^qEDn`$m%7IVjD=&o1=ANX(hMT9=CHKKJ`nFM_|}>_I-RxMlyVq^ z!~ga;j_tyui>2?}+)4t_sfc|sny&zQQP`hES?B^{QL7^bi}^RUK%^vhRbOrTO@FzR z)(H3q^E|XS0}%s0m69vV*q^a9eq8}EyU*I}8RQ(P7UOM=A%vd0?>|DKY|~hH9x6WP zwU9T>6Z*{EGY*J$DV)YhDiOsVksFVG^Et0k8iBVYyi^yz(i8Golp7ngd%jWx-Qin) zhe-mxRCXbkgiR{4mq=~9^XQ}ShIUX_^#A{C`JPk)v1 zIV#64FwY+*Q*Q2`2f0p2fOj8|K__048Ieujt*C>|%uzGdLmhJ6G^#8STPB}1cGU%! zbtHC0GI2Vjly(x)Z^PO9rY@%&Z+}ZFQ43a13W&TOfJny=;>g!hdDG(TK|785IElAAhG^~nr;qF#B# zgUBr(G4P4nDu;{qek|$bGQ=4${#!Q!DP5u z5ti8bejUXok=b)_ONhnt;N zlFlKpKT+_*bs5u4|5DMZoRKrN-c|XDvEF`VIrn%(kYUJnVP79Ci4RtOU6-!$BsTxU zjCc2O9#tpo;uSgRe%bBMIF7toZjZ#@i`WVnJt{n2-k`dmY#8&tj?Sdj!el;g)h>c) z8AaK7L{~SJ`6R6~ksd!=ppz zZ5_wImf4Y!&98k+%ue@TGnzlD9r-$Jey04{?oIQFnTS%@+0u=9KMF6kU0)D%Gu>YZ z<{dH!J>SeLTTxsA#+O<#%I|tFFkHqP=vSK`1?B^yY{btrH!n$4gf7h=<8kVyj0>gF zTE@u|x)GTtR~pin1M!#J=?IFTl@oLt5u!FZ+`)o4B#cwcWXs5|uPCwi`z`X!Ds zor;T0OJy7l-TqLT4R+6VImYCtx*Y4Li(Fj07Wr?@^I2aFsfik`S?QU0bwpeKx>acw zH)gTOq>iGQy0v(c+0bW$HMdx<0WwY+BKwzT`@fcHINXrg%N{s$YtC|q;ydGKc+qK4 zS?UjmYd7Aksfa}lY$pYZRx2076BOczsnYO{6aS+%CqxqEutf)4Xd`gpROjr7{Vlrl)!D0e818T*Hqib3Gn z2?bZct&NND4@%mS@-)sG8tB%Pb9+BR4?emu-EEM<>tfQ{f%6_~RH52e+Ft?TSRq-M z4-fE_s#9K&UeX>V&ljJMnRb##t}Hc#Vv~yGN<`f7T*17et)NU~xmvcwd9F?f)GH#F z20pF2m8+sxc*aRnAYgN}`HvA?BeGugOdyHqkm51J$P!$sXAFiy+iRkjUqsKbiLl!g zU8ZC$7K;nqzO>dF()LywD0h6iq>yUdXHV9@xk*pX&?aT2ck|D(hW=`<%{io}wIy1q zyDFo8KQ~nYcyftRM{W}GtMrqZNO-Eo?oFbX6#3{vuUGQU@7HLwPGa)gdz4XX>pE6KQlv&TtMJ>gM*H2dJAm-LVP5DG zPV-*Kh24OFUaB9Aon6E2ezPc75(o}Lf4h?C@Ye;}4YLcb8$Ogql+DfvKPcdxJmtZj zYNjmM^KNe0z)m|GUTiU(vg}_tN#}WabRf2{bxTx2z6ZKuiumGAo4fnhhdVxryEFRE zx+8nv##v6Va=74G+;4?BhO*mB0Wi=;fal>NT_TOaHAWHMw$QeALII=;TNeQ&T8o)cDYe7_I7g8;$)+BYe!gwTF-s2Zk*EM z)@{5sx1!vDLh1QZ7iN);OL4PgPt(Ibc8|I$*ggfz_^J}JY#{&c%q3I)i_brm%4MC_ z{OQsq=p*EaUSrMkvK`5*{Hkik9C&&ycYUzBD|v0vsjZy-_3pYjk8Tc*h6pMjbjEDR ziA&HAM3nAS%>6^#vBoaFs<^LWe8Izmy~-ZsOtG~5>cZkd(TEwq4SJ*ZEuXuqX&M?4 zxYs2Y@dELV{4P99P_0RegHJg?%vOUU{%vu_L;bf|+r^uE-%jwx^njDz@KSnIKy7`h z%QTA%OJpB*4a-$psUDSL6M#zb1Y*$voC*3C?SU8s*(i}iMDadJ6Pe57WNpv;u3FNdxcDb%uEwiD;`-!Y{0B9>Ku;sGg Date: Wed, 19 Jun 2019 10:33:41 +0000 Subject: [PATCH 05/40] number of executors in examining cluster --- .../examining_cluster.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index 38f9d167..fb59d441 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -42,4 +42,12 @@ Go back to the Summary tab in your EMR cluster page, and you will see links to t * **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ * **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ 3. Go back to the Summary page and click the **Resource Manager** link.\ -4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ \ No newline at end of file +4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ + + +### Number of executors in the cluster +With 80 Spot Units in the Task Instance Fleet, EMR launched either 20 * xlarge (one executor) or 10 * 2xlarge instances (2 executors), so the Task Instance Fleet provides 20 executors / containers to the cluster.\ +The Core Instance Fleet launched one xlarge instance, able to run one executor. +{{%expand "Question: Did you see more than 21 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} +Your Task / Application running on the Spark cluster was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. +{{% /expand%}} \ No newline at end of file From 12c3d89f7d61c882334addc4b123bfbe6afb5de9 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 19 Jun 2019 14:09:04 +0000 Subject: [PATCH 06/40] revamping instance selection page --- .../selecting_instance_types.md | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 34b1698e..2b734e88 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -11,7 +11,9 @@ We determined that in order to maximize usage of R4 instance types, we will subm We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ However, at the time of writing, when looking at the EU (Ireland) region in the Spot Instance advisor, the r5.2xlarge instance type is showing an interruption rate of >20%.\ -Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. At the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge +Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. As an example, at the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge + +![Spot Instance Advisor](/images/running-emr-spark-apps-on-spot/spotinstanceadvisor1.png) {{% notice note %}} Spot Instance interruption rates are dynamic, the above just provides a real world example from a specific time and would probably be different when you are performing this workshop. @@ -19,6 +21,22 @@ Spot Instance interruption rates are dynamic, the above just provides a real wor To keep our flexibility in place and be able to provide multiple instance types for our EMR cluster, we need to make sure that our executor size will be under the EMR YARN limitation that we saw in the previous step, -**Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 18 GB of RAM, and also have low Spot interruption rates (maximum 10-15%). +**Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 30+ GB of RAM, and also have low Spot interruption rates (maximum 10-15%). + +{{%expand "Click here to see a hint for the task" %}} +Instance types with sufficient Memory and vCPUs for our executor size, as well as suitable for our desired vCPU:Mem ratio, and are also under the default memory EMR limitations:\ + +**Recommended for the workshop:**\ +- r4.xlarge and larger\ +- r5.xlarge and larger\ +- r5a.xlarge and larger\ +- r5d.xlarge and larger\ +- i3.xlarge and larger\ + +**Previous generation instance types:**\ +- r3.xlarge and larger\ +- i2.xlarge and larger\ +you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take more time to complete.\ +Also note that not all instance types exist in all regions. +{{% /expand%}} -![Spot Instance Advisor](/images/running-emr-spark-apps-on-spot/spotinstanceadvisor1.png) \ No newline at end of file From 6289fef5c982b77bd22b6bd109b46d277b2c2ae9 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 19 Jun 2019 18:39:22 +0000 Subject: [PATCH 07/40] notice for port 800 in examining cluster page --- .../examining_cluster.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index fb59d441..b8fdf91d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -36,13 +36,17 @@ To allow access to your IP address to reach the EMR web interfaces via EC2 Secur 5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ 6. Click **Save**. +{{% notice note %}} +While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which is not allowed for outbound traffic on every Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) +{{% /notice %}} + Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ 1. Click **Ganglia** to open the web interface.\ 2. Have a look around. Take notice of the heatmap (**Server Load Distribution**). Notable graphs are:\ * **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ * **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ 3. Go back to the Summary page and click the **Resource Manager** link.\ -4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ +4. On the left pane, click **Nodes**, and in the node table, you should see the number of containers that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ ### Number of executors in the cluster From 1170f24f2b0aa4f2652b45609be969a66a80c522 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sat, 22 Jun 2019 12:08:47 +0000 Subject: [PATCH 08/40] modifications --- .../conclusions_and_cleanup.md | 3 +-- .../examining_cluster.md | 15 ++++++++---- .../fleet_config_options.md | 2 +- .../launching_emr_cluster-1.md | 23 ++++++++++++------ .../launching_emr_cluster-2.md | 4 +-- .../right_sizing_executors.md | 4 +-- .../visualizing_costs.md | 5 ++++ .../emrinstancefleets-task1.png | Bin 45510 -> 0 bytes .../emrinstancefleets-task2.png | Bin 0 -> 50232 bytes .../sparksubmitstep.png | Bin 15091 -> 0 bytes .../sparksubmitstep1.png | Bin 0 -> 14975 bytes 11 files changed, 36 insertions(+), 20 deletions(-) delete mode 100644 static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png create mode 100644 static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png delete mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png create mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 84e98dcd..a4111f7e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -7,11 +7,10 @@ weight: 150 #### Cleanup -1. Our EMR cluster has already been terminated after the Spark application we submitted finished running. Just to be on the safe side, you can visit the EMR console and check that the cluster is in the **Terminated** state. +1. Our EMR cluster has already been terminated after the Spark application we submitted finished running. Just to be on the safe side, or if you didn't use **Auto-terminate cluster after the last step is completed** you can visit the EMR console and check that the cluster is in the **Terminated** state. If it isn't, then you can termintae it from the console. 2. Delete the VPC you deployed via CloudFormation, by going to the CloudFormation service in the AWS Management Console, selecting the VPC stack (default name is Quick-Start-VPC) and click the Delete option. Make sure that the deletion has completed successfully (this should take around 1 minute), the status of the stack will be DELETE_COMPLETE (the stack will move to the Deleted list of stacks). 3. Delete your S3 bucket from the AWS Management Console - choose the bucket from the list of buckets and hit the Delete button. This approach will also empty the bucket and delete all existing objects in the bucket. 4. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**. - #### Thank you We hope this workshop was educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR in order to optimize your costs.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index b8fdf91d..d1b77a57 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -5,8 +5,13 @@ weight: 95 In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. +### EMR Management Console +To get started, let's check that your EMR cluster and Spark application are running. +1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running. +2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. + {{% notice note %}} -Do not expect to see full utilization of vCPUs and Memory on the EC2 instances, the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very intensive and is just used for demo purposes. {{% /notice %}} ### Using CloudWatch Metrics @@ -25,7 +30,7 @@ The recommended approach to connect to the web interfaces running on our EMR clu For the purpose of this workshop, and since we started our EMR cluster in a VPC public subnet, we can allow access in the EC2 Security Group in order to reach the TCP ports on which the web interfaces are listening on. {{% notice warning %}} -Normally you would not run EMR in a public subnet and open TCP access to the master instance, this is just for educational purposes. +Normally you would not run EMR in a public subnet and open TCP access to the master instance, this faster approach is just used for the purpose of the workshop. {{% /notice %}} To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ @@ -50,8 +55,8 @@ Go back to the Summary tab in your EMR cluster page, and you will see links to t ### Number of executors in the cluster -With 80 Spot Units in the Task Instance Fleet, EMR launched either 20 * xlarge (one executor) or 10 * 2xlarge instances (2 executors), so the Task Instance Fleet provides 20 executors / containers to the cluster.\ +With 40 Spot Units in the Task Instance Fleet, EMR launched either 10 * xlarge (running one executor) or 5 * 2xlarge instances (running 2 executors), so the Task Instance Fleet provides 10 executors / containers to the cluster.\ The Core Instance Fleet launched one xlarge instance, able to run one executor. -{{%expand "Question: Did you see more than 21 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} -Your Task / Application running on the Spark cluster was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. +{{%expand "Question: Did you see more than 11 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} +Your Spark application was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. {{% /expand%}} \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 3af88678..065696a8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -3,7 +3,7 @@ title: "Fleet configuration options" weight: 85 --- -While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the job is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 1aff90ab..5f063e15 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -13,24 +13,31 @@ To launch the cluster, follow these steps:\ 1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark** and **Ganglia** (we will use it later to monitor our cluster)\ 1. Under "**Add steps (Optional)**" -> Step type drop down menu, select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: **--executor-memory 18G --executor-cores 4** (make sure you have two '-' signs)\ +* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Use these settings (make sure you have two '-' chars):\ +``` +--executor-memory 18G --executor-cores 4 +``` * **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** ```python - import sys - from pyspark.sql import SparkSession - spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() - df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") - df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) - exit() +import sys +from pyspark.sql import SparkSession +spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() +df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") +df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) +exit() ``` Then add the location of the file under the **Application location** field, i.e: s3://\/\\ * **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ * **Action on failure**: Leave this on *Continue* and click **Add** to save the step. -![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) +![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png) Check the **Auto-terminate cluster after the last step is completed** option. Since we are looking to run a transient cluster just for running our Spark application, this will terminate the cluster once our submitted step (Spark Application) has completed. +{{% notice note %}} +If you are not running through the workshop in one sitting, then don't use **Auto-terminate cluster after the last step is completed**, otherwise your cluster will be terminated before you examine it, later in the workshop. +{{% /notice %}} + Click **Next** to continue setting up the EMR cluster and move from "**Step 1: Software and steps**"" to "**Step 2: Hardware**". diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index ec53617b..0ab9130b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -22,8 +22,8 @@ Under the core node type, Click **Add / remove instance types to fleet** and sel #### **Task Instance Fleet**: Our task nodes will only run Spark executors and no HDFS DataNodes, so this is a great fit for scaling out and increasing the parallelization of our application's execution, to achieve faster execution times. Under the task node type, Click **Add / remove instance types to fleet** and select the 5 instance types you noted before as suitable for our executor size and that had suitable interruption rates in the Spot Instance Advisor.\ -Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **80 Spot units** in order to run 20 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 20 * r4.xlarge / r5.xlarge / i3.xlarge **or** 10 * r5.2xlarge / r4.2xlarge. -![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png) +Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **40 Spot units** in order to run 10 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 10 * r4.xlarge / r5.xlarge / i3.xlarge **or** 5 * r5.2xlarge / r4.2xlarge in EMR Task Instance Fleet. +![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png) click **Next** to continue to the next steps of launching your EMR cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 2396e85f..9465f7fa 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -17,7 +17,7 @@ If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid g EMR by default places limits on executor sizes in two different ways, this is in order to avoid having the executor consume too much memory and interfere with the operating system and other processes running on the instance. -1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the default YARN configuration options. +1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html#emr-hadoop-task-jvm) in the default YARN configuration options. Let's have a look at a few examples of instances that have our approximate vCPU:Mem ratio:\ r4.xlarge: yarn.scheduler.maximum-allocation-mb 23424\ r4.2xlarge: yarn.scheduler.maximum-allocation-mb 54272\ @@ -25,7 +25,7 @@ r5.xlarge: yarn.scheduler.maximum-allocation-mb 24576\ 2. With the Spark on YARN configuration option which was [introduced in EMR version 5.22] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew-history.html#emr-5220-whatsnew): spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting ) -So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio), as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. +So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio) as vCPU and Memory grows lineraly within family sizes. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index dd073980..9c37530e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -14,6 +14,11 @@ In Step 4 of the EMR cluster launch, we tagged the cluster with the following Ta ### Analyzing costs with AWS Cost Explorer [AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. You can analyze cost and usage data, both at a high level (e.g. how much did I pay for EMR) and for highly-specific requests (e.g. Cost for a specific instance type in a specific account with a specific tag). +{{% notice note %}} +If the Name tag Key was not enabled as a Cost Allocation Tag, you will not be able to filter/group according to it in Cost Explorer, but you can still gather data like cost for the EMR service, instance types, etc. +{{% /notice %}} + + Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ 2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ diff --git a/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png b/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png deleted file mode 100644 index 5932f55edf06d66f76a03a4c8a6e50b11954516d..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 45510 zcma&O2UJsC*De|r1w=qZL_t9WK}0D+Kq-lUfFK>Ds1S<4OK2*+geXmsrgTCJQjCc7 zUJ|8BGYY{-m)?WaKthtUpRo{(@^2SKLp^OG3WXCEJKpSd=rQy_NyT$pu8 z*}GR_kIAJq_(y7{vvxP^w4d(ni@E)DEBg^q5+M?yy6Nn}y(>)~X%OLHspe`r!+t|) zL+iC?LF2;1p&=UWX95W#@8aBqZ+;jQy;MpnUS6*1`;f4bHN3ZvRAvP%1A(3gC!m@A z0Dsv;UgBZ@^Ol2!dcRTVuCOPAaey07nAo$fcyu=-OLQT%=g{9D6~PtN`zKWy zVbrfW_`UqOp=jSTFh`5)7|BTG{BZs-0bp2l8v~Xqo1RRC1<$_K z8C$^qxoi}g>S$`wcnnlz*RGT6@XuCxlaGqp&%niure&1RyOJ`4Fx1T-;DuRR29eRl zDpQjJMgim711@YVmu@5u5TEwdJST!;H891c=_#@609T0y5k7wv&bwCS>vRU`X{)Nia?$h7bLWk?epp3!$0n73T%v! zk+TSjd_v|-1Q=Ab_*@EJax?k_?#gC#DIZc2(wrU}wUWB8RnI`uCrM-oot55r3Rqxz zM2B%7uk!88{|jef07ZA09bZVQ6d?qt;QN9HsS2xPTE z_Snp+p>abfEnb_p7pdH;wb$r5@#{~UXlnP3C=-o8wKdDIHq8dr0gMV{11NrorJTp2^r{(2;bbCq#N^5W&oi0EJg;^f^$H#kF)GJ#oi+z3MI6ydv?R)w%3d z9;=t`upObGflD>GnMhFE_R8P(D)e!8HJGsy$zoalrwt7e!Ttb6W*c8; z`0NDGZHFI~?T(Vqt*!4C#lKLjT)BU{U=r%nNFMbFq}aayL;u>$uv)EKM^Yxd+Jk0p zEz!dQ%aSIVu;PNbH;XjMqYag0FmLN!SyAD@*K>tv?b08kHmX-rw{RarxDs8~M-1Bv~kUU{mSZ291U$iSOitsj;^7^pbmzgz%6h!3d z6GJKU#lvMg+#u)Mt#3YfvVbM-FCp%rC!1<<@8w-z$(|>iI3|~2yzm=IaE-4yE~391 z>JW`Cs*C&Qv2jvDWciF-kr&w2n788@;1Rr^yh4lf(-W#Ja%^D?=ydYI(f7^RKg8_W zm~gGNGIgdmtKjPKD(~^~Yg97YHR$@jG8A>)V|=6kWgv}Mj%9acW)80og6H z!DZFv$UN^(UrUad9Mm~Sh)~p97oG-FtUm718yhAAN}d>w7;Yf$3ACSBlSvOkZRQ_` z4=FomwNWtKhn6llECk=K{m1` z1Cin>TO%j!kE-|E3GQHjPDoWB(t}4UQ^QdD0)zm2{ed93Z}r;wOBfys-uvE_q|_C3 z6co>C_u83<@JhI4L!R8uk6ndS?@ak!HV86&WG%XIkWltWyx@y!HN#1=kbaLB)R)Bx zFKt?jWABJmcIbfjj<>Cru_$H4N_L&!Z>*5# zOC;Bw5h!PAD>rZPwfOhQO%+b7!{bH`3){BSq1R-Hj%}z}U*7ed=jw5<> z-W>aE-Ywh>F;xAo{f3bsErVo-k_y6=AEe8zWa zI^*CRjj_?D9-r|DTALIh9CWG~Eiqov!ao!-k@lWh-chjTN9~Zn6OO3o1)OH6 zH-!tLrjWh6c(2i3%}Youd(`%h9D0fJ5ntzVU4ilAb7|3HDW{Y`Bl-i-gu)nN>pFR$ z-*S8qPYMJkRw|nVWTtL>1O$qIBuLn^a~`w_QCoGUlI;9(7xJ0y)=t4xjfuqYd&H~O};9t8HF+ZgKc^t{99%p$% zxsfvM?=!o+ru)kQQgojUJ`kjH6nq4+7Jmrz+=+*9I%K^I`jk6+Kb>JZZD~t=Kh4j7 z1oV94r#R-^1Mqb3nocfUJ&s#r`xNQ^5`jf}Y!~;0-GY=p@Ufe2cAEPIxOKy*8B{ z9z_vI01PSQAzWenX&WUmvmr8;j6`$79jrv9-1LaQZJVb2YHPhY9Wn~rUT@2>UO`mECjz4T5sru$z*Sl z^yID2sfE^+HCA=s(%GjB;sfp|=Uy6k&lV+F`zyFPHhS951`CUY0!8}0H_Sd>`V;dj_V z!gO+VEk8skH~CS+^MZ`4_HG?BjMITttA95(9?an_!~eDO-C4pStg%7!;4mcYas{2! zXm#|nU@nT^t9e-yMP0{DBL0K}$r6K`@l|bOU#8Z+Uu9y03pY^YbDdNu`(x;f9wO0- z>*?cn0`YUcH6hRQ!{eTKKH5O9B=6!$HXgl>2rO(3+{R0__~{p^zldLJ%+5m>mqCP& z`#1%if(a5m4_2S@l(By?RuXc)mZ-ckLtwc?6{-I-7wYsj^h|d>>b_>J{>Tr!D-|b0 zCvH2rFA}%ieh0>%gCY&>Ys}XI-ZOUUYl6WMh{(;|bNsgaEo+L0Odq_*i3?d0Q8UB1 zLL2|fsY=8}Vr*J$wsZnsj>>ouu=Z*pd?qg(RbK4e<`F>l1IM|uaxhUqeCMCn0yLN{ zms^Lr0h?idRJw!F3Qevb8k z9F^}Bj&k*20RxankLwIU?4Hx$eMLX5_y>uZ$G~*&p&gmom9)^>pfAMtX&`b^io&UZ zW~$oyhDcllxMFg1tBdiQm$a;eTchuP!<{MXQfQAJ`7Nfze(ZpyG~>HdQq$|{K=`Hf zwB9=kxnJ+)cCmZ-z^KPgboK5B{UpB&K!d!j3bui!<6(s16nK>?kkH-0$^g2k#dMdl-!|0^cn5rPk)9CzSXNT#?#Sv2y7T4LkF>Ld z8&dFT|AF=t1)K8IU%SwdhOL`YuHx7Ae>bvQrMC}B{PH_oCgIh=?)4(0PaAHUIW(w$ zseiUf&Ut!q^kLShf&O4{vfL|Pcrdb|Rjlru=6IP^>H3f%HlMyBoe-IG@R5EZJ}IY! zb$GUk6TaykiG4Y+^>e2izqI2tV4+s>n(IdB#<0Jm@hyoK?D|f6Y+M8O0&adm4;N8Z zxAZV5r@?YEZSHXW*XkcRW&&RKyF?f*Q~&S~fVCN0m6!lok`?EK$DSoLr$?#1V94nA%uj9}cn|?4TOsZ_1QGXX=HqS#`Stl7nl%g0HJtBFbDF zLJjUvvNRfma|22ULPdWG1G}svEMS|Hyg1>a=GWSc;U4j16X8q&N_rwKC$mEM?bQ&k zxXrfQ0|H{(k(;_+a0@|c>0s`zKiLZ2JmOB{vx@rD6p7*fTQMt@TL&z4s_gm_IX0Po zIEAOXU48BT8QqVAJSWwnDyJXAHj=j7FHOYVbC=>!tW>0q_IAEg!kiO$AJV6RL67{* zdy7oHI(B#W9$q)1qGD}3DfEH@Mz^zUoAU5GUWC81P>{7Lck=y(t`bra_s;Ge%C%KQ z0T~6cY?_2wjuB@Pf2KjIs#U#?kY7iLm@$FNsK0G!hYdF59O${D2+m&*m!Ox*4L>gB z9t0&3SXAhahCv-M$2~UPI*oT~yC~rT`@Ey;!3@^1K^RSnyq8OB;C6Gk%Zp2jN>T9U z6}Jz4CV4z*-8}v4eQ{5@bd$|3_ssEv$`4Hvpsy?X5=EE0)hr}_I4;B6*#nJQf=&r{ zxi1P_7mRs<)+TgG=DPPb%-hc8_yo@jd8C^L8&08`6L#bbattonDg_-JFT+hOH1w_f zh|dr0EZi*K-pOD2XFX3)Czx?WMDNBi6{>+%{>b;Tr1HaGA(NC9HjwgWQAQl|TB}#w zZ!x1F%%7E`wUO@K>$z}zw}v1hbn{2SpYa>43^wzTpT-EIkf~k>WM3BC) zB0Ff^xuGM7O*0je&3b4Zb(J@Fp-g2er=hl2%=_x&%i2IPcSkMP)EF$efBgOO@)f%? zY;e-$Fabs2s+}3L5G0iFu*1c^2ab+T%!ls3U>%ze=cc@w|GBoQQwYjBxiO=d#3dPT zy0!7{R!NaAL_q6q_)Hpdd!!_BgJA;ivf2C?lF4|JTuR@e=H&z!V(>yLeaNq^<*F|xnY8+GUw$L6`Za!(4YOgz+RI8|sMqT_dK8tKh8v}jp!CnQZ797eKpF(P z?Wu2^ubKKJbe~)HZidanMtFo`|6$Bi)?x*Nl|n!{?B(#ZneozbVr*u-p5~po-yVHCjM*cF|glIENpEc$E|lYl`x|zII1nkwO>gRwJO2b zJ32aMP83-VZGS3y8wa@Ap9Cd33B|w+uG1&F#?xd%J3J73{`yv1(cTLwmoj%@Ht zxYic1NYRB0UFmqpcU<}Eik-~)!z+@_NPlptZM^D&b@1-afx*aOf+mp`lEWF3& z$t6sfoixcWYg>+EhfjAU*TTxx4TMlYj{n%L71eRewnPWntm{TU%%{#NC@46ppk%Ep zUyagM1jqv`c_b|k=POMAs!2+l*pU|QyL&}ctmJJ97)KYxEkUU53*&-NhSo+9NU8ve z5|grhJl6{_cSY_d3Ecnaw{R@0mP}yK&J~f*ACG#JJ=QX4jP#}zXSBW+XyGCJogZ%? z&(Zwo`dAe6h5c!l&8@n*+S(q!-L^mgAn^Cr`FWinre!BJ)n-2OCKU%du8j(t;6uir zs61R4PkFqAy+EGpy;oWBtuzVS?gB$1+KHX^Hx$m_P;psxUU*gl?4s#n9myyEe7fS*v{6jW%q*FZh&$Txir7!T%N z3iM(D-xO)R6Y|f14VPYP1buZP+Zjq2TJ!H3_^q||Mcve&jn{9y#u?cXj*^XcwL7Je z?s*~bSY&ZUyevcmU z{G9_sRDEI*O=gXA|13emU&!S$C+M1Mh;h{~RKn$qnf|VKtvbn#GPIO_WL?~G>pthw zhyOrEPS<`9tte?J#V$wv-J^}b9yNT`#sU^Gb`8%MwHzNFm~HVJ8~mu$)74+h!?%|{ zBaVLK=0BcdHSs9vlb9TNkVVg6_?zv7%V>_Y)KT-9c1ZU4xWS0jkZW$rdIJ0BS*+X4 zBd+#K)Q}zd4=oVNsDNhJS$3y)0jg182R{3_1K;<1*$PW~0vo$^xbc??(4ZNYj+H|o z-#PSOlj7F?0fsW`WKDFvlfDul#$#_d% z!mE153yYWRXD+~=2=6bR^Jl0`E`_dW;?@|JFBQ1mJsY9)+4p0(gHarR87pX8^zSxy zRQU#MV2L{?F6#|kEbuYBu}^-XX%uL}TXMx!)-D_+R{p0I5y5w*a5EXA6KY6t)p(<^ z9~Pm)YOQ6aa0%1hdMU4~NoI9(A&xV3``30egfg-sA}sS-*#Fdqs1YUK2$dE5CP9q$ zM*`|AO#9VFS8+AW4fT3@IW9s0h+d%c=l^a%!O^Dx<#$u$s+9OC#VgS$-4G9_NN$zX zEJ7w)g8j9FN{VqlhvBlifxyjuZ#|JH&$jJu}um^6Ie>@{P@>e|`m1vgBrs+;QtP;U`AOowBhRhMIIJnGFoe=D91erDNYr|Eyvq_G=J1}hwetKY7seo63ZGD6H(LC|lZ zUH$$*>+q0;Im?4gX8;U_A52KnAXh~&Ut&L9#JF1Y?s>nKa_PwnBl2!9Ui@4OJ09vI z=o+{rv~w%+WE)vZNUEmJL2b$`Fr>$Egg57`K$H}V2Ge)&AWa60Zp*IRJG2g)z%|EbyvYE8G&IrB&C zi`EEs($aT?reLK@9j^ZHZ9_zR^rcl!wHL|0W6D4T(>+>U`HWXO#XKj$>mjL5A=Lg_ zC3V&Mkb)R((d=}u<<1WYcKEELp;z$e=$H)j&-2lcKk9%dZ$#7uK!CQN@2DGxpIgz0 zdFN~@tnnf!&>|0D7FB@W&J$kG;vh+Z(FylQg0Y7MvMzqd73%k3)ML}{yG zwYC}P2}=++m;Am~fOpb=lAPchWFtfqm2n_AW%9}_;OoP0V29I3>ym;!z&^t%(xD90^5k}38WSv zkeP<@5xeW%5jh@eN`bXYTjH1utS<*V2f3Q8;;VP7YQdB5n@V~UX&}Rl^n?GN)5`@a z{g(k_>ywL+ClY{8gSVN#=OL7=Ya7cO?Vhvo9~Bhz%&(wnc-IdDSjT!CqO(M$BP&krtPtS95p&N+=PK13VUPlwa_YsWh~i<0aH=zDTUAN@p zy#Ek5ZZw#bSAY#k>9eiL2`Gh@NqSkb>^JrTD(CYV=34rfnj3CWiL8{Hn;9QbS+D(O zV>4B`%h!k)%C;F>NUf}b-CRfS`dbFCV{r@PeDe0zEkVZl^n3Kkm43P5*}Y%v_UZq0 zGN!%6Gg!w;Vc&N?l}^n3I#_-=$j_huOm*~zdR(j<_L64V{4d+pVFPH4Phppdr-THC zeP%N5)3_1LXwId6#G|3YD=hg%f^sCE;??hgPhBd6+fs_+6|!`9KT5%YG;yf*UY?=5 z5@#Mf58x(m>al>+E5u*j^2eUk9|I`F$nOq(>iv&XG!n$?t%nh1%9vZ?HRz(u?Z|R1 zBd#ukV{HWkSu2c>j4AOX$a&c>Ci3>#>Q?eYXz^pxOWr>mv*Vgz<3A#lC<~6i3=NIV z9+(Wqc~kWD7>X7x*P**2pI?16)W0(D-Tv;$R(_mD2dhMPSAqUsWf;1XMRy;b`;@sn z>-m|!K?E~A+_e0SG*Rw!$|7sjS7jmbvxdgY4sf=Oe}7tJW37)Gee7h|A|IH(&=E-V zYYGy(vf1U>QY?hbDJ4co4Unc6XeUee+z=`T$Y#oN(ukgLaP!Y7tajb|&c0a7^HTRn z>#GxSif5~6DM3JBzNt9iDIg90V(Q*?OurVkza>f5-98Ey$@-xl7a=nvK0Bm4BOxAL=V4kU1PJQw3OjffBbNrGKxt5~QD z%mV+5y@`cSPPIHwQvis4$TMgkVUd9^Iwv0MTi4{ssrSDdi?|v?!>-hb-(6y0%u1>R zMPV)j&irr0r~}$(R((Qj8o8abGHY2$xE0)lQE%&2Nx@k{+(Q#yi;vfuP(`Xw>zwRc zaTg9BDMLENa_?-tFT)Yhpm z)Iv^hsfR?OV~vMCfQq0mZ;BVYEcxf8)3y&I8GgS__&y%+-@^Z$nXx(9{ z6X(M03@q#V>)_qO zgzI1*&^sj8{GT81AKfxPZ0t?$uN%9k1oJh5HZ1_DW{Hn9h`^jxbGD1TdpAF%<dz3MYO1s5$ao-Bk6=NPl?hQG^j7 zj`_qkDQ{5+$uRe(y!8%ccVovoKd=Gs}gSg>m8J{a*A4`jTrP&a4{o4=X!<(Z?fM)I}hyRE#! zQH!!25%LBXJZ_s#lpO;b{23_q+3=8<0$_(gunCCFNz;N6rpAwU-%UN_cli^HOes#8`t5psw0YU1A z0z{}z)rF2VxA$`ApoK4&)m3PY^8j<*M^BRd-1>P586Vep<#FL`0;D zGC1Fh#09KTp_>p|MpysSRsoPBbAcwF{LBhcwv%K5!>8BDa|dT{JpO3hLv%hA=2a&| zi12HCChT4!hH>plT#4i~(hzKnHw`2@S3<=xuIny&vhg)&aT&a(klLByU*xk~@W-v5 z5{veg>H4!(*R44(K9pY~bPZgby{K-GUb2k+RerfvaF_cep`G5Yy9Lxs>`JA+^t0e0 z#Ex_E6Mj{E>$I5+R_4-a&IZ7(gP?l9?tN}Ji}3z~$je7Qe0U|*t9>9CIbn6vU&>>0IKK2%Gf6tvn`c^S@INN#HuF@%v5Lft4agcXWI>EwtpCiNmWIlO!YcfjGUIoyZe)98a93C<<;dTayK{Nb27}1w;EKFe72U#a`?OU#i zF>lwUZK$xLx7X9=czaPyga6;ou=6D91~na?t=Ow54+!1d0NWUuy<`v8Z;7mR3@os0 zJZ*{FuEu@erDp7H)XrvA8`dB{G*}e8xL=|al2z7_FkPIb%9eCr!2@^j8Wq4D)0neF zf>wW6$4@*Qwe${c8KXZeNoXmPKybN^Mgyv54!eU`){nWua-fQ00d)H{`j}O4WclOB z)#a?Ev$iH}ibM}Uz%PqK&rf))Kz1?K(iqqM8P~g~I@T}s962FqEavh4i^l0AQ3~eg z06};iW8w0NyZGB~Q_?$mZVB1|uyIUdf(B~R&@r&(F4kG&EW7ips8L1p zX~I`7O^{_J%bdTRW;fHx7wyvV@;y}ue(pZ$%3II2` zs8|g&IF2orZxR1g#d*^qx9Ma@)V=Xv)gHy6R)JScIpEX9>9vz7!Eqk7U-aGcB@>AL zAKl?(6LvTubSLo2|5K#FOq@&9!USq}M%du<(dc7?gGaePNVD7xkEB#6$jT-$6xhDD zhJ`rk`ja$H700?lp%P25>zHUNhxlp9WEWrO0jfTgZ(RjeGdO{qoA4gcp2%_O#69^Y z5{BB&H!``lkZ%Oy0@(8ZorTWQi%JMCB~3jA5R>CtoA)nV;=JBS`;#qWbJoO5MaZkY zF|pi1pDH1Pd8_Pz05p7SW))AN3@@EhhaFGt#VeOg^+p_u1x&ZQj$lHz2%#0791lvf z(>pbCTsi2b=YB*>!D&J@@4f~mPl7CNe&y)-q1q&nD5tJ=HEt7KwB~K;GFsHw*|WY? zt{%5$CjE2Lw#-6mnEJ}YZw^pVBj{UWr+NzJlrd9y?;~f!0eVd~ILN6zD}}`Q+2q{u zdtna7`Ql^?{+j`EkF6HNj=ZhD5dOj(eCM1eaz{PxxbwRS8QwD+kR#{9+m3;Q^yAJJ z6M9A=KRZiL5(2vAKmGR_1}mm62fF!!i`gG?IsN^s+%>+H?8Ay0|J@%m02J|`LuUWuN z-hT@b#wKawlZT`%gFYPs$@&9d%4#ebN#aeGCV*!^`mDeUVlRqxjs1m8AUg_^F=Syv z5D=*A1<;)KMNCHO|BEd^tG58?_ne0fUNkdZ*h^}STqe!o%FNG+z`B~?d@H|4 z`o>!nt-6MaDaGc$l=5DC9YE@qu1VATI~J;ND*!6-txNQ_(E2YXDXg41dfov4aE?R0 zf9gmE>T)lpj zV#jb)$Xq{wgo;&JZg>O=vMH$dlN_dpQ`Z~a)_mEk-W9Pw++8M8?z35Etleu?3MPde zzh1DqIx#fuOUWVyjdBZzRXZIy^cL4uT8e#2aKduR2YPkZF1fHdl%f_+mnqTQmT&6k z4sKC9wcWx6a479h?{vu$&G*Amt#DbWbpJ}p`hL3ipQ; zp!;vZ9n;#Lzt)DK#0yNmXs6G1>AGH77f%Snu?vkn7b)xg8iu0OV7af>ms6j4D7o$? z+4*{Aoftj&UIjV^i*@6zs)B;$iey08!> z5q09i{4H}5blv5>zgGWJ49W9X_O>4eU9_;j=#8*ei9(cR%r*ay%Dn3J=7ZNwA^SIg z#iS9so*XKi!VJwQU9!V;+Q&#vjv&(!rEVGht5osLLUfUzhnE`tfr{(5-loQWG3yiC z`DUIv%Tk|$7FWD0^y55iJ*nX0<}-wAx^?$g{k8ZBU;O}FJRYLcA9*WeQPTb8ad=P; zeh)qWyL1)NA0d0H@2gIZL!cz)t((sU5V||l`CoZsW@MIZxCA&=Ahup(0Xs^(2))se zu5irWbOdUGm&j;Ykl`V;KO0Xgm6q&1avF&UHgHP3<^w&>`y|-bGS}ibywWeKCxoVR zk|1^{TWCgEZGs~|Cme_o{rv1*d|*=K+CEapV9NJh?$Y-yIs7Cbqps9$QqGBT!rxgL z0{+kXzw{&?!taQ){DeK-0{v4nvnB?5TkIHTiTrou5c%N?!JoJ;Okd?`sLfRH=e4o1 z%@E@lPiSRn4zPV-VSK`o!nA>h;4|X#uqDAbA@zD{=QGOAzVJ!9&9 zFb|>nK<^6mTI$=-ZOPYjA=~VqE7{<`m{+_vlSY>l$Gl~Cx)DuG8kk7<@v9qi#b{H| zxw2`tWrKl* zUtRZ?xIT#BaI%3WAts#%B<@@O>?*hpD-8#YFKB%Iu7}^D#%u`eoG&>`NQP1rws4eW zhzzu7dLCT=};}+>M{%sH3#?$8v_m*$f4|{|9st;(~&sE1kBq#U=h4a=zp6*6(TLHfc>Z3?i}4BP_+80t zqpDwuP7_Mt|B~4%*flJbWab@qJAUY|k|W1U=<`|Oggf5}J4=(p+>PfM(9htsf8b$k zDvjLksLtuM4@U*ilEYAaE&puI-UO1zlH$IwAY{m>M=v<$2D@164u2Ew)FLhb!g6Lj z^q0@WsRb>YwLsmw4wQ=viZzRjSKp(+jV)8?_PPuBgq>xe5cV)lpu^Egw z>H4Hn|BdM_@_ttVebL@uvNXL{Bj;?yti1{UY0e1o$RXLfoWNfM9_rSje2~-IVm!fQ zrPs!PNq^~PYE6i7Qpxcjd0M>6i|bMxc{&0X?QcX0e0Y4A0NLW^k8F&SqG>`rgya=tk<;1_l`(Q(!riYbNJd;- z+VPu=y3%;9rIb9Q5Ba?~0$!j%J{UVy=#I~U#MpheYt4<5Vy&C_!ZI#*CNj0w;!NB7 zgLVH=<>bNye*bZJVsiZK$dI_Ju^ESUwE`yk#Y?PW4sUp}M!jRlZ}Huyk?IES@@r-m zIup*Aa64x{0ELIJ4K$x4R^FHBPqpZmza|dA zM}Rh1a{VEr(y{yRo;I);IPo8a(;^mq>}K--{Eb4pm6kE@kyBJPuv~!a@(MfL9cn2| zcucOpV?4Oq1a=59_SxeD!&Lxv!b4&hkFSw}FOOceAc$qk#7y;le=Sh6GxYcMU;g{+ z^)=#sjT@DITRr~ric@63Sd>GKQsApZ1yazZ{ZP6368d6Dh6Z_K~zb)AyfbY-iL~rN=SbUhc+bOSqdM z#=HOl%or-!?2Z$VIIH=uJ|sAL^ z$-27?U*@Ep1*CJIy{h`U1#a2}aII0EaN{H;poG`)U-k8`!UGg+fwnLHHwg^*yr}-K zNbi^RcbGumXPET*q{0!+ob``cgi7tZlF(Uou7mY%3qLY9!@Hsn3mZ+!l+Dz z-Pq(n0tbDr0Y{itmP(s^v=5`|k=4vlDCjw}t;K&SeZ@1_vCRYb+dnIq_HK0Smk&S8 zC=~Kp5lACWwGs{sOlhLJJmg*WZI59jIJy*PY>53_8o(ZLIqU|$-^mW8W>}kcL_xx-idOq(1}J=qofCk7jw5s!3X3x z$S&&*yJH|WfKRhJxLjVfrJH*j$^6oWEqvNn< z!kPl^U?2#!mFc%5bM0Xn#fcpRvM9^I`0;i)SB}wbxv4e{2hx_fT-y3GzVnOhD-OuM z*YS#&K>JY-clV|hdzsM#cqWI9Ya_1jr!ZlWYBb4vIZs_t-O{PQ2Jz9Z`j5~RMA^{V z%QuA|kAZ3An{nLm1HDqJ1Dm7xg%oRF@V~1-4G0LlFkeCr_!_^X184K1LYLn|?D%|IrZW9|#uM8eBA)$RAWxJK!m{LX(K33jJn_T_?_{bm+Y#74p%4a+z( zV$l0e=aGuI^JX?`wp0ABV1?)}23s+cjA^}w`7K zZGq0kXqRVkKbs}a@|yqR17D5ZnA99Z==8UofcFLS1R`(TROjv-!&%l_EY_x>2wmh* zTk=G)xz4JiP{FJ1zy(C?=E|BCSBef6>M_~0>j0f|fmte+=3#e7-r_Ng#@Uv##)&bb znmC`nZJfmZtu+o1kTJ@iK{hPH#B>*@koFjuDj%IczD$Z%XaaW>`6;*r%ZWkIw9!6& z$K<+YbUpe6&jSh}XncKgPh~{^whjw;>CVH!q!j@{!Y`miGePOXf==$Xn$uM0`7o44 zKq$hu9Wt^YRESqucT9$Pbv?t3ksT{w=N}G}*9GTV{#gQ_D~%&Nj}$}l<%pjE#iKNd z7?I`Dd4x$t0hV=RU}6R!)6|1jlM&b(j@`6?0qqhfcH{0YO$J>MKTr{1bMq&yvhUBm z6QJk$BDk2O$tvf=BY-t0(PB}c!b%J${NOlB6%+lDIq^%!Go5O?kAfrpx+1P!XXF$g z2pN$@))V0Q=3TcJ(Zj^JaxwvCi%}88&xbJH(q|D~lWz69v8fGzC)-?bhB?a^@D*MI4hbA&n*j6`sEy0KEBFh54IVD*ov|22@XS*(yhD#9KUDtlci{Yd-?cY zJZGFfO@6BKf)kh91QN*zplt2jQQ+p+W(AWr38*jr(m0-Cudtq<_*-PnEPy)=f}3rz9-Z{TomBp(+f4*3d|sGPP*zn?#*xS1 zXcY41ehSMr3)apreYl!QUC(*Zko!QeL(Z)etipAX$y&$Yz7k|m<_B8&p&vS6K2!V- z2a!CxGcrH7`OVz(X_qtL-x7$UO1w6LKk8+8f8weSfj$>!Os;I&e$lUQTlbof|IP8Kedtw ztt;(p8^ga+^vT_-02yllJpfJ@4Pqt#LGHIJ&&c4b<4RQzZp7dSnt>%UjLW5^Nwvpu z?xR?RO=Tc9Dp#oYZ@BNEom=+FNZq}#8HsiOzyyvHfKDC%;6w}oSpBTPbH58NVs1fL zz-AX2FRFr}C(!Xg()BEeFTPXeDhOnFBMdd8Y3!YFAJTXf&CJKR){0T_^bq@FP8+|lI8h7>81bNvJ$2?dAb3|AEZ*^{My#s zJ+3b=t*)khQ!%VHvPK>tx;BCH8tb_S*qPF-mF446U8^IWK?Vcg);^NlgG&h7Pu9CO zh2-&9pOT83=J36+aP_HS=N(QpMjIRL!J7PuvJ_rMBhVJ;beKR%^vwn}AYarE>M!`g zdh#J@b>c!Wse+w6RO@y4Z7bc_>heO}i_d5K>jA~J>z?CKu|Adt);TBE_dNE?@4epi zg&7~W^=3qEq02b+5!c^JXV6V97O)jZ<+P(qo$OPONOpUH`merIR(LUm%#%t}TAD=3 zq@}80%h63MSo-5Gw^TN%%Gx!=UW&B%opb4XdI;CW0nswt%(`SMp9O#hudtpp2{@y4 zP!p6DB8;l2!s8*V}nh)>&HsxN1;tMYoeEr?ih}n^J+*!!uyr?;em|>Pr846sFM%j}ZeN4}h+ad^tl~sN^b@z?g?(`M- zXP~1nn6Dbb=3__0h8Wls-z5y10ehLJjzV7zojO0SzR`VOTh)K@S20^(W z(mC%(`g@%&VI)#t=)X$KJQ2lS$Sz%}>-n2%%zo$K&Z8X}eYlk4Po;gsj0z8Nya6&U zM{XA}Nc)!KqLkVZY)Q|0+z{DOHX2%~Sp0wY{}XPL9>hZ^8!u%$Uv5EH{;@ ziy~G1Injq$Uh%L&lr_+iX}jPoHKiZa^^dp7)};pDh>T-VwmS2`P60j92go+J8$aL@ z^)0*_Z;kY3@cH7;Ifn08V4`g-5@g)Oe?)saV6#fO{SpQFBy#Wj>BkO4fH?$E&(v;} zZcD{$`8B_d^HA(geR8pS3J<2eqU3~Q`G<4{$%bSKhV}eZ`au>9X3oa-Y#f1_0TQjX z2zvX|EAbn@5oZ8L*cq=GWK&^KR^}CSo@0$;XO%+FrnqG)PTx$SUtd&S1wYtqWS*9f z2UMf^d*V-m1{8tgLb?xboD+Kb$tviQb;fyNy-}iUw}ED=gK+TI{SI`|nX`9-`uY$f z-E}{SbtU@CN=cigP))K`C>LeF7-ErQq9m<|el>FPcBv9=_3gNsuE$s}UL|EbSRo)c z@+p`oHuHTV=DaYyPBerDVT<+^f?!-%V!Q6=j=B_lBl?~mWx@p#fI$0FbD!+%!ip8T zJ=%50Y$aK)yyO^em&v%Y{P~j1`Un4ryxq-cXEiZNa^d>Hi|CT&m)YNmE6EhNy5>)d zGw{2N;GFQpl#kt^H+9Gr5NU+EL5UeQ%kPS+s}tFLVEnxrgs>15D&6^=IaGnAOFIV8 zxyb=j`Z7w>$b|liJR%QsHi3#EJA}Ce=0W;d%5914N0gPM=#t#+ zCGWP!HJRz7w1IZKuslsqq8TyZowVD3C&=FJ#J1S|-Io%@c`fcG)AjAO`q0(cQQIW_ z;uNPal>7NJ+bxgDbE7|JOxf7X8^(=t)lb9j?F9Fn4hzr18eI;o%}ZDc%rgplR86No zQpC7^lZlT@#EJ*Qq?WvEs94;v+a(-UFs%SGx%;m9lc+}-feB=uGy~Z_p>D zd0BkT6E9Il{}7B(4E1Q*rL!^+PGyWIh~VtNYz1VFN>i5=_RUY=+&<{MJb+1TAL^g> zr-TXUr?RjX{&v4|&06qO;^KG5V4&USk{~L#6g^W0 zl;5P>Na8B2hk}!^Awm^*0vD z+W%jx$$>}g)BsfY`A$o~7WnUJ6(;_hpnhEUPQy{atpdy^;s6EjGYj*~b(0Kz&T@=8 z!FZvFiMEd`DUq~cHe^R2J#6f5kDdC<7@b~?%pyZ_Cmw0i zEcBlS>^e#HQVX?XKP`s8h2C8MIkE4s|C1CqMXB_DM)XT^W2z$N4PMAQYFplIYujV~ z60s|Pi@DnAr5Ac5x0Rt2{Q>dFT2}x6QPx4S|Eedz?m9usA@9A>Zbj{xNbK*=ApfxY z?_^H;F+vt#F+}EYSaCBetxkg4Zx`9Esk^=mn3eZD>^~nt)sq~*YP?i2>_}FfojXYsT*L*VIILZK8HTvG`Kxb^}pDA z^LQxRzJFY$MG{vkg~*jgrLtuYp~X(wvrP7oE&DQ5NXnM6har1*vNKAS5F^{zm7T$4 z8N-a>dz{mC-}iN0_kG>Z^L(G*_x1Yyar>i*IcMfL&pFOx`MkG@+v!U|UFo*N%d?ap z9|P>uwmdXy9Sk_+-&_q@Ay{E`B?IQ6C?Yd}<|qE6dpG@I1)E!Cp8 z=^tM+FgI*%u%9uHYSSdKZ;8)mF`7;k%WX9XepFDb9SeeWAgrDSuYclXJ8UE8xhv&f zk^(Dt2=R2YSY3I$`8#mw;Fdx+qu;6~@oTu$Vm)$ZeV%wyw_8Es6X4r|A0B$Jjd(Od zTVK2Imd>+eGSsr~rD&09rp2>l?h7xMw4{~#_VW+`xORS9ytU=o_0iiWrCpD*^(ve( zlIdBABEO(WC}=P@%MwLntYkY%Fm4}s3lSJvfn%L>tM~NIVdtEiBV>vxKsWqBiqcpA zIDB^yMJ|FpzqTT2t)~ce!^G=QH|(aB(M|rs7eYFuAtjbm`mqs|bFQWYBeZ8j4%-|> zSEpy<3MC4fHQQ>uL1F6>p=>ek$U$L%)z}S2*A>Py-F|j@>$B!%zWu;RqQESK2mQl> zvv<2GBWsk=*Jl{9AcP|93EdXAMdow~kKsuPWDw8TmAqV!5c4ee=IfE_QtO z3xLZ2!koTGh54%1r-QC}Cl)^MeQJI}`480M_ya6;^$L?nHU4Puqq(y0Iy^M|Mo|!%nkNCBdy*@J5Vc@1ttf8|Ui+YMOZv%P7iJ<;d1G*80K--Q`hB1xTAWCDWiKREJ~i`c4>P;)p3g@b z5^@d*3pgsGUUA?ajlc(%8`vmo0uTa1y;xxC=v`biaAMV}|SyWwxMg$(W})zvI3v9no=GQ=i4xL9ts4>nx5 zy%j~~2Ffy~&bgG^4Y5wltM^hSD&*A`YzaEUZ@SzDQf2@Uw)XTTti@*75rA58;&n^q zESP}%1y@Z{=Emp74rv=561Qhxiwssy>*1~(OksyLpH0Z{TD{> zym0yP%f8c^I{8*A8l@LRQ`=1I@m~3K_ee4e$8n3zpRw`p2lsQgqTj#XFL?$YyX=2@ zHgoAI8BZ!C_LLMUwH*+5oz&?K=)2Tjkaxs&@$#i4i4dkwp#v~8<(g457jcg%`Lg>-sC5~M5j6m5IC(THE zKEWZks<7awwfsTF6}^X|jQ3BjON9a_4T0r_FL?EFzv89mXcin`7>y3~T~m!P+56JP zKqUL_$M&uk|J(b86)R2s1FuCX5RnlrkG=z^k8J)=BkuR{6Q4iSg$}O2OF;KsE1?c8 z50aqq+vU*N?+O)4=V%WC9?;$dn-|@^@xFXt44);$82>Uk90vfoi)?0w8t0Jf{MNB_ zo^dI-VDD-|;FCmo7{?)iaEdqs9+6;Sw~se z${XE2?d*A$Fo4QTc9K_<;E(Bf~y%S6-hb2DjtT0g;I0|TPgj<0THwc=& z{_#60&Ap{_Sl7{dpqdsL9bckqw%TS_$LV04}5QmHUQP5-x&e}vxsHnWCkkTlzzwW>v- zcc9i1&nbKsApV?rs3_j4rzw{Am(g1>-t#lL%kx%5h(Q+J33(FEKwuukLJNBpd$oq1 z}4$nr!GCh2C%1Rt&kzorCdQt_=^AHzJ_Fpl=YCzp___2Y3g(Nw{Hpc1G(q&By zs68(B|D04Wb}wIA(B}jK(wh8Hwx*KFdq-`St6yb_j?5t4-%zX_?wsf~95&j1^9p0- zhqX(pS+GApnd5*C2-cXuSsPm+FfEC^T&fZ= zV0pX(ED^n7FJ3#|j*V?#t-6hZ29qy)pryZxMo+-CO9O|H+hP|Prbnml;zd8ov++;4 zp4QjUtnVkv?q_j>7c9X#e2?XIh9XVaCR}EN$hf^h=<@Co4fiW~Mc-eJGRnA5J|vd9 zMOMq9v`5jLtx`=+>f{0JER1`OG;%$tjWSDFO#ca%k^_ae=6d4(NH{YbBqz|BaL2t4 zzuvTZPXp{5Kp`CP5}{d47`fck$frxsc5*%aJ>XBrX>F)ZDa)!Z4UKtim$`K(#=5({ zQ7kz%cPDEktXKCwp$j`}8qYdhJF;P`<#{DrKhi#T0DqXdw-$IsC8Ot-3)y zazW+8lBW^k{>ALpq0^Vv2%NGSNLm-K5Zt;4Ux?RH9e~t|mCj#9bNIosf6pDGY67?B zAkks}e&|au1e4rrLB$Z{9pQ@|JShoDAuq?)I?UY{%OqT019YI=7>aiNdOz-VFd)`mj)UcAn7na@H2~0&|;B*$Kra!LW zKgH64Uho@5@)>Kq1`O#Tb#aMuDinPBDtEmJtVY?`wiAoEI+~U*Nw0W-^mU%Moj|D9 z%WQUg_*Uj!EmJS8Nytm&uH3^ecTR-aG2lO8&$^|o9xqZGzwD;&pKyD3li1^zY#PR>wwnW5_t5bVd%)CheLcK$5kcP6ut4>&cVPOu!Z*aW4nAm6S&U#<2yF{W%05|k<}!bnF-xK}d4z&XUh}s3 zk!{Ia7#P1*YOsA>YdHsnH)^}3om%uA%QBw8=4 z>P4S_mQafqdH92Qyvb?2MabHLYTqNT<=x!8*1b z4pRep*&A5e^dhs0B35 zU|e9Wndb=ouB;wmDxdIuje)vdqxofr`Cdfm7&&6mTq24vj9tW%s9wE#Y)}zFP6K(#=UVIY^F|j;o5IBgi!vCrCzN7L zHIO)Z8zsak;1D_AQbcWc=^Zfn8OUmA8a-u}i?|$jH}z$`bUqx*(_-rBj1bShOSt^` zPVSdGs08QkB3Ldo4*_C7Ak@|MKbJK6rp4eweH&Y_qNv37+OXzYWiUa1fh4#znh^-v zS9lb35>JpXz+!^>%)j*z{=Ka*zq9B_XMfD_OE-t(o%*-$jW=}q&eu*xtur+ps9# z=G+N9p|Sod)F^*wR2c=L)%jq{0rh=BUlV>bsD!);UVDnW{(hZ<7l}Jn8*<2In*fTd z3F2e4N2BT7JXg;YRJof8?H4>?bXHTyW_#QAN)GM?+>bgZeOQNR)(tmx5AqTeO~o5u z2iu6nVEb0NUK|x^sB&uZFVGy4sc|p65>D9iEyDs0p@IHz&Oq#%y_NgdX29gsMKSKL z7L*Du47C;Y-Kczl4R~-NVuj!Ab+0}rPZg$V2YugZ0zRJ&e}N-{frz5ShYrZ97&L$Gd;l zY}aj@m7CRKTV~@ZyEkgx8mYyoOtAD5Dj#{7;4_f}8}xswa2(cT#O5ZW!nCIG;k{(_ z72(%cqxOrQL>l(F*_n=Zef+>QT3#rLyAx-QT|O6bi!1LXfty!>)N(PzS$AwP7uTV; zW!vYhuptlce(cJp_)wdNqCwCKVU-yVQqt2q(VV(V$X-xsPGu`~G{U-VR%k6NdlihT z`u-B%kWBXdROTRpU*eQ+3*1-+f#iiowUKHDSn)E4G4I92f0 zBRS#F%(UPy9XzYuvjR^Qud9!nNq=X2j!$)1OOPBj1)v|4c%L)z4$P#4^4= zSwdl@F^?a_{${IYk(dvAc;PM~{7n5~?bxh-b$H-MteHE?OXZ8l#sTfE(yqmU=*~V& zR(-(@aIqQ4=w`FW%O7{^puVmN70&6IEA985ggFuD%9B3eS`C~U3?9HrUnlfh60RW? zgtqKM;>AcCCuiN5dq`48$K_2TheJ}AbuVLyHaz2ZcRgZT&E#%#Pd0il2O z$txAMCGm_E9ar|%aW>F8{{3Kx;46EaBBc|5>h@@(aQu(xg-d!~rq{brT%!RxXA0Ma zZEI5W{!6E`mu40PoW56RTXESFXD6`>;#TumGmKC_@z<7~72Z`usqBj?Y@72^ve8c7 zHUZt~fgvS!d*rUl??BZ(n#?CD2iCL`J{(o;HQh2!jA|+IX^1l_%`TPr$SE#h9238O zJ383A*dZrb(QimsDwuCWUI7s;Qn~~Ali^NC12?=_cO+#-Z#Y)NfmOkRh$I*cJv3&^ z@V;VGaQe`J=PiXJR)w>|MzxHDkhN8kP#F1pyQgj8Yr)a5I&8OdA2A^YhrOAOxfn%W z32sqZg1)*n2xy>3me@aLgE>)GLv} z@qibKB?_=F})u-VrFs7$hPK!+Gp>DKIEpDr9Aebz8}yM-sQ~lDY-J6;Dq$< zUW)gAZ=&Ryw=L|+iNo-CiUfO%&aLp)?vUVYrtzRfcPJss)7ta6JJ_#d@}gk?gOrOh zKjOCWrs#HADxAk;e*^+eprX0_2xvYikBeJAqSowmnfu9w03M#%Ed|H7{!!u)(u{>R zP?32C4DFRI1xtdhiHG%rUl*Xo?qafzYpE)s8*-{84nya65iAN`a8|^-C^$tWgtK&* z2&=Asy&wmng?YeV2|R!?I~dw;#*@av{U$gh)&z*D`HVF{m&ijbg>UPwKt}sCYY=Hh zX#F>ns^%oX(f7s|qA`#yJ8&!qZxRo7;ZNeXf;h+A;M?tmmBhN;eRdkmpIG(vHG%y) zSZ!U0@XHXNNqoUpxCCVxj~GF0P3VsF2l=%G2&2T@(H?*}Nb|&#+8U3Q=3MFYs@)y( z=s}rH+hyYJkMI#@)UGL8f9>QM{109`B`kCe;tqndhz|y=r_ehRK|P7GFoy+ z{c^1Tb6Wy@H+0+ug_u}d4n^(0{Rw@65BNDn*;VD|mn2af$MWmenpVMX`4honb_(qD zu!&$-obz~m=Y`h{?)mT_r0gnhS*qlN8%Kt^x+SfjmnM^N=g=ZYasRYP9r+Xix$FJi zyK*H>?Vrpmp@juMC$z5M|AjNbCDuQZKNU=2S%ZwOuH@0;YaU;H93y(qGz1vA=MTd1 zDXvG!O?6J@r9*dRKcFjHF7gAHQLUi|GaK6O<<~q5&k)xa zLRXziZEo(Do~fj}Fg5hkz0{mn@eAeJ(|elbrLn}p@~n2HJ$+)u>Q?UV$PFdeA**S! z*guH(d1_8+f1?u8P5>71nY5&mc)0v|!fJ7kFJHtJteDVav37oU&eM?ZZH1S@J_gF|L z{3C#p)aZuhp>vS+Qd;+z65;*b!`9s^!P|E#Uw(`qallB$&58l5m+Xo@eA^gS<riVc}2h9Q=Hz~ET8sOo@8nMwFZ-X;s!7e0dDU;+MI~U zl-trX>*W=79hB95jFlY+IoG*5@`v4D%|cVP9`)|O`u>6qjKFm~JC@ilIzhH>X#3>~ z-Gvf&J=5Fw*n_|`Qf7E(*f&T9V@HnID9)8J^p5B|pMQDtpVpB+BA*v2=dB1ksW4`t8wv_uZ9)%UQcCs3vFV^z&Gru|62x{Kif9 zsz~tfZtFArKs*MX{m=@iDG)^2QYg`?LvIpos>iI``%oa8i#mQd0<7@E$QIuHF6R<{Lhlj>ctT$W&bUvH#0(Ez8Xn{`Dh*+?w~F_I z2tXB`HM|w{7Rs;PbnjwFm1XW@c1vRyV$QJqgrw{rRWdk$e33eF>6NAIX9>17ul$@> zwifZ#`Odnd`BRD7um8T6I7`JQ6aN{T>;aw{)YKP>+&~R^Dsy;Xz?eO0$L%t&O&EW+ z;-8bPK#O^y77wJr;GWKI_^?ZiPgM-Y4(wIcD6NU}D4z!8Pj91!ylm@+m^xCK?+G~_ zOY$*%M{-Sy9Q{Q}P;m4}lFq6b+xQf8)^6~|jVXL_Zmt8nYrMlJm9aO)cBIqK;&Nxi zn{cYT2@*$pX4kjT#b1X~nhP`U0vQyA&Y>?QvxxOexbm6=&RcrMkj}ch7#KZ-KqT3( zz9Wi_;n(NGrL7+q7;3(Q)=hDOP-_8bNN258^b2gu=^kraO>5ei!?Pff2cL0*B`KM{ z`FQ`sR~ONS!>S1sZoMQno=+l;^PP-D?Ea$&y9|re<@3LSmV%H*N(x zr08rs)@KK6H3jwyk0tNHt^4HRD9D0g?eTEo-RZIvOU3GZ(|X?Ol-uQykab4V=IWR4 z^6Dm%S{glTK7bo=(A@%W3k!Jt;I{PY@pffd@aZ`4;Xgn9{;ZOv9;JkHl|g}%DS;p(|calqw6vS6;D=k2GUhZ73Qk*0%IRZc?E~kyt9tAG?!%r!&Xzh;mf2{|Wzu?KCwL*Nz!v&+d<&xQ|?X&roaxFX(ev6$oZqmb8O7OSY_?>iQa3ZmimHqiw5{^ zf5q_QE&U*2pLigRq?v(~fFi(eZ#IBpzyG#$D;}1#m0sr_S#?tb*=M%*Tvd60eGy;o zv6o2m*DBTIp;Zf5b{lzAvE5}{P9KXgAFqgCSL}->2w1nr;?CY6Rxt^_?{=o_nAYB= zdVEZWOLuG+N@rl&!>x(ctX|TDl;&hz60!UUWwz>klw!N9#wXM)7-xR65~{o2epXqW zbc*ldL3UdT4kl|Rna9BL+JMK#m^<6G&0zdQCEhIYnkXo)*}fqI)`|-PVdqM4w}?;Y z%N#Xl?`f^HpIsbd@MA2_Pd%GrD;jX2eSI*I*KQLl&+fl7n^Gnl`bav_8m;01;|hfo zYk!Gh2LB@q3S22*+Ov=5TwID}rh$Wo(78YCi=6L3&oT9?wEs&V*bj+Nf$R_!^q>jb z2S_R*kbn(H4aENG3E*-habXnNs6Cknqc?a2BLvVRQ~yIP=>7lePk9-$oRZUONhMJJ zqka9G<-2~AFmJRTq&A@WumBJqh9<|r6lfx$;pc;D4_9w`UMooPT+AeIElCkIwyQjn zRwvY$ry9?~`nN!cqXFJswga-LEwD};bgqaj3E~*0LwsJa?sOPJfON;5>e?}CL6PPg zH)Nrl^CRuMZpZuY%8zUH?hpko(y6XvQm@n59CpR6wR3d>!;BD#@*iFCu&x}V+2WT} z>`#pqjJbL&y>n+X*}c{Sg%!W0>3lYqRlRgTv|}Nu+|KAuxlI zgaF}xyg?V7v{l%?aGNM8vMi@_ioc^3UYXRs3vAP#WI#e*MUC{(t<96r`983N0ImZ% zH8%yc1b8hjQ*_@hUW!Q%y|%8nq07;`zJ3}ys>l~2Y-gmvd_%GgPWKeVF%K&09sTEhbuSUt(Tdosyg%4(2xFXAcWyt)CinYYDRlBpT5EC-b0(-H)J~?X-t0 zKtBGl>K@b8j+a#uUgffs{79n{X029>Pfs4L8H#Bk$)y}CJYBVL%iIn}zA&mOrehTN zm1SyEBZapIxj8gd!zSZx*l6n`Wpn+D9=B_MOAg)54SC1!Kws%v@ugIuxr}a1$)Y>V zS8aF~u~MJOH;q*;n8$utV;Az>VfO^eWzBYRDhAxj5;hnx-~O~8YT5@NG*RB58%jVsb)2uvf2O0&LecxkBYgu76s73Mf zNQ9BQHqlV<-A?gjY?3Y_M%I^;-oLPAS(*?Rx3b9aZq+Jvu$VU~7p3lJ^i(?Q2kQnR zv}I~Udz%WtEuR77MS{xVVY&eg4W!C9!c@N4_bdF!K?PNtQm+KX^mrf9qY_S+H%vZO zj4~%ifs6#Ye#*<~>NApCVRDEJ^ov!!x!E+5keJ$5fb<)m3RzXp)j~WMZunV&y02Ox zAMe3^IsPFiPCudFW0b=yw!VLjQGTQXyWc9oiWzNwFOw(LUp6$k62^@F1^Y6Cr#U)! zA@w+k6;}3X$%T39o+sj%18n?0rZT!k4h53A5ERUR5yqrk&*7b1jAju}4uikOzhRL3 zQm2z_OI%#Fnl2=|HhcHaMl~Bmb2j~KMjU8Hc{QEUJFB-Rc^-Q^-Q#Fqr89Huc-!eS zQIi7emS>){U?}4u`<58Rf41`S-!lw=N+KnDKZErytNyt56Q<6uygtvHFY@7GfkrIINIfrnjOo|I%u<|l>C9Gs2+w12YS(M%%0~HA6Da~c zg%&X6d{y8^OrjRK)zeK?^lEYbpz!R=NIJ#)R@*zYmQi1i#v?1nrpU%ip8Dn_M~z%3m%*d=^l zA@K14n&*O{KgllA^wbw#8qq;;D*WU`=yXg$~O=@{;kw-f$*^9uFT{; zPi+$EtLuAWS7=(@?o+1or~Q*Cu)Mo=b67>aJ?1C}iYr)t(x?F0pJU+_Sf->u(utj( z?(E_3t*xHY#}bbOCH4G@;4tokj`{-VeU9Nq!xoX^CrXZ;GeS7MO;*2{oe<>{sOjo@ z96CWwu;gKqu@{<;n7AO!$(QiSCSfY=E|n9eC;D?a^qU1F^F|%#ZljdTjM=ih0jC?- zh^Q}H)o&I~c!Kw#>4(-#D!s2n;m*|Hi-q2c__}(MwEI>r{2a7xr0G8e9nmGT{iB>b z0rzdH11S&>{S!_>P+>Qu>Y!4HXkasJe(oK9z4PvV_{rsVmimcu>1|1nsX+}-gbw>D zdHnoJ)R@54K_QOgr8hsqo$Ggl{HBsOw^@dZX3pR}g5cJg)zB3GQ%`wwn;^Q=kJvoN zSOxdO*7)$BOAVps?`)GCwlkqTC3lDIZ5AjlI=;hBI1o=LK`RMxtbP4x$`&>$mKP826pB-(-SR;>DgDN6c;7WEBsc zu4YO(4!ibIRrpC!>QnLZGF*W$8F#w(npxU%?zNM@KEEh0t14kH%X~n)*vq;q!#_3N;t4=pu$dErylaGL3x-^(#`Nj;(`+&?Ospv=*p>k z9+!Ryw5W=4VVA3Ui)*W30jFM8p2p5kA*&p^3i)w*E`qPTV;o$ zN!vnmZ7yf8Eevu*0A*g=%@vO$wvCq>ShsL$K!(>HeQjdN!LNqV+IPJf^YW)Ql)4Iq zvSXNv_b9T^g!IsIOZ_pVMP*JTA^czyxqMpS{d~hIxSt#1{PEL@LFZk&S>{_DPrX+u zdOOTPd=wk|{Mtkre)ltZ-HJoBg^{^`yJL;7GTdi9q-Hk>?@~`l9pBi^l}|V*B09LB&ssd6*;98l`g9=pJ<*D04N7h~v&Y^fspC<~S=|*#97y1u!d1p!I39 z>W}u^s|gW2l4B2b@;&zPTHc4vDL#8Upf;VR^%IZOXh0!-E`RQ^GykqR3lt~4UkO5I z)T_b=3jJw?P1s%h%s^=RW%8qge>T^&I`VH~+o1cW6@rL)Rev~7w{ACNhtu@=BPG~W z*hQeI*8yGE&TL?lnX7D5z4LC;=6XhLO%OF$usy8RQKOW9VKBz%pUKtD9uxHN4cQL3 zq1O=QTT8`gmh_rFXV+)c>_{=9r;HDu|{tRRYT$#jNEJ&QEJMGo*#4AqYAF(T| z&c_-zeVfcOHjWmGWiELHh!AYzbmqn5Z0RjprNC0UxgbF4zYj?AfYFHx%CmmB_FYcc zN2BF_YS?8a1)btQ!jaaFUpKg7c$VIH>)rCuT(%Op*1DYfW&Dmve>B-{=rrhXNUAwz zE@=))E4(U@RZ+KOw;apq@xe}dGlkLGeiUtRos)W9q1zh=Te;;WIIcNnF#baOG}nW! z$fA!2MK>y!dTzYvHs%D(U&<7(C?{_| z9yllwSg6wUcmFMQ&{hGmbUr^7NB*1o=ub&1V5o-bKmS6A@&{oG;G01E7hvy_BsUnh zx&p>>cZ1o2YJku-VHSR*tbF zRl^*rUloeh)D0WCuefLiiEdPr+$hfq99#(;EpEhC)e0xlgsE1Tl>sn8bsF^%T#EN% zc=U*GV=)uev_aqT@gK??elt}j=Vc$RxJn==p-*kP^emzEQXq$_V`&N5*j2+>#YR?D z)QQnVIq`+^K!p07vO(O`8)ta^oJfzEvOP6rg!##rvw870zyvXNy=o5F!czj5uGW5) z;r#Tsi$%3B?i>T>M|^$-Su`-m(nb%ItHdn}yjcGJKsi z<->jFWM9j0;Is*wYc6##!`E&sGWa0JCOB@`nl#y`i|W-O4NR7atP59530qp$PUz`B z_Ol||zyBDwRM=aCD`+f!?bhR%LlrP4^w#jodz`MY;I)U-Ffl zN=NHBl-1GM1pz@z4PKfv`{Cofa)yJ#9WUd1hV8vqLI8ECZ75n$+8mes=8OXr8uGi7 zLT)A@^rg&r7LRE){`m0xs5MBL-_?K8jR;UU$SDvZVA1CVQe9p>207vz{1wCYgs`V(s8! zh8k-l&X}^Lo8X*4>|FtQmZ{|?;Y)Z>wQeaY8D6<_2+%A*XQ%qtiN204YE+v$f9O>5 zKo*z65&i=1mQuj2W>Ms4-WNE4y8LjS;nLw*pft48IBqdJV>c&lPCVx`&`m38_KbBD z&YaDjvhn%Y)5S-bm(o?*J5^tL5rLO zuTmF-VrpPY6gaW>Gp)4R!nBplRw1phZ%N@Z2Id0+5N&XPv?_Y(M8+3gp)ZIm!qu=NBg1j3h41DKRL}iGqGqWQ zL~|_6CHI<~V+5`6rIeiK7gUqxi49-QXIsyo5I~2<9}tc%5oq2oEU3tcbc>G&qN`4> z%rbxvn6!h^47@+!*o+=m_E+o{YSLBd&@pOK2&v%NMg1$KRbuXj8k(p4FW-woO zV1rmcAlh7K^jz07Hybw-1HR5RNhv{eo`SD2eUFRR?u1AmiVq2F(uimtl z<2Z9T-~$(YtN4)yvb?60k79gRe}@Q>8NX3k@$y@{V$>*pp7A5f9e6TGr;-J>k|}X zHi+7A(~|8d8*Qqu+Vt6QAMLNoUDkQrc`5zR!7RTCxxj)Uh|%08=7^2X^AwnHU&;*C zm)}37S?q1f48%I6R3cw+dUkUnx~B}QGPPL1)#JB-+6fYV(5o}R@=cp#Z+v+z18qWz zrMtk&qjvYvvW@Bh{kuH@SG)hW?*@I8Xt{CTLHnTm5b-ayw|qt;-H-$*r`i@P##rCdlt&y8jkGN z7*pjS_%dIAzenCU_Vs>Yf-Qnm%h7~54HSBdJRcWE#7>uUsk!@B)?am;v*o=)?}}|W z%vLZ!`Qa=Y<|bpylf%2N-?v15J)+4@d>@WL*vmaDz;rlgkNAyY)L45B>x;gZ!}XBi z(qQWHN)DIC#+I245EJw}|JmgPDlEtCgKG3b^TsOMkIRZoWYv|(u&qEvi}9Q2S2Pu+@m?pIHJUD}ou5Eh)?_E>0#BYIskYYn(My_(( zDoqo7+*tEBhx?IaY+qLNvxK~)R6+PIFdbC9k=0VE7EWx)o_*j7V$6izEKlH8tU!j< zwyFlQ)}FTzB|zm7CWcFc1?B@I>X!g?=*KQ18^t`eklj9e-wn2A&qOE0S5P-QQUMdf`vHc{L4GV2Kr^w5Zitca&^cPxf&w5Gg_+^6& zI8yu~Sp+GaQ(Ynrv)g)nhIUSxMeezdSHU8P+l5l z7+#9EfzBT`8gans1fW0nkH-fiu3u#)nco{yfekyqLw@NF(AEdJZ)AJ_12QW{>_kKe za;CY{*V)e5PhK{(ps+B%V5i_R5*W+B2{Ds6ja^C{t@)1UDi=OZ>$3Em$s%6*jdKRW zuGb5*u3h%`grhbm*Xxl@Vn#@-?(5?j&k{CVyFxOyh*=m(F*?)rmXu$`Z$sWNPpYVc zXPazCZ8<15TK;>DIaEdZuSAhQ0mGl!C=q`b$bWG8-;Dp@WB>0aj{nvRA2i!n-%(F` zoTau@Ia_M>5vUinXZdEgTI9}zqxG&*5v>QnLZ|5Va<#Hmb;L}O3B%8QAGHGB9C<>x zf;b{y;s=bcBPlHIa(Zh69Svnh+06clK=}Bm6XVZXb3e#Xed1VeI&ej={uOpW@^Rfx)kUaIUk1j!^NqnKcX1}f2s)cpT%(C}38eX_ ziQ|J_Bb9xcTztN6=)6>Yeb}jy>p+PaL%HfW--*#(a`~v*GizB-Z>hu!350<~0aX+I z;Fjv;aL&L>_jvC|hO+~^eUIxPZfrERTY#-GUG}aqA9UzXbaU`5_j8V|g{b7NG|Fd9 zUG>Yne(M+eqnANeWKRAR)D@`90nqe-^l>50&9T$G&tYdMqx`V!VArx-XVYO|L8!aC zNgZ&g7OSAitYK3hU$Mw^M?JqHaBYQb{o@{;jUdP%mt*_lG8?`n_adAZ(l7%~u#0z44dT{%$e*%OvC|M{*z3FE z$abo4AEf;NTN3aWElSC^DLxhIJ3}rI=lToY1#j+9QP3OU!cY(VtJMef$v-Sx9v~nd zQ9h7TXo(C#K1F5OS*4|L3B1^#v;}2_c%~hIh!x1klc3mIkjs0Dqtv!$URY5Qsy;3i z-(&k&$Jy=;SsDq$<_8$1Q0YFz=hfxu8u0H4i(APxHoJoHo}q+9-AOoSYnh_VVr4Ie zO_g-L+z&M6LjV}MRU7#ynShDhX<-vNp^w*V+*^{pA2lG#mfqY*FOoT~)4HXsYW&7Y zm8U(U*e1|1h5Z7c{wl;5MAWoVj;tD(Nc`}b=i#$qNh+Eb@zy4#E zp%&O@Z>;)mu9axOJSlu>-B#d3A1r|Gf6~06vUHf`UKCH+VP7t~7W5Z|1L4$VCm~qZ%jE(T@ZTW3l4`%(l#@`bcLdb?S*fF53st)w1%8 z2U#vIu9aeBi9#Eya9)Z4hO3H?9MU~o2X@`7&aorApN*BJ3Q_%CwvNT)M4xUAwe`SX zGS*5b%Y1oSa+jj;IoddX9y=F{8A5v8N^5|xC8l?`Cb$LpAxFxde12*1^3KxpXc^Ub zaX9zx?%Q^jN#|R}4kV6992pK(t8Rc|)fTcfLM3cF{X)FLMc-c^!rs`t6?3%XQ8`i* zoHojcL`d%WLIo(fm=t-6r4zGNHZu6EFv3s$u~-x*``tF^Zg;a@pNwEE|24 zcqjVFu`-jEU#|xhTwktO;@Km20pHxP98~Tyj_@R@ypnyc!@)wSO&?oWyI5y`z@KRt zZ#dWUVBC8d6Micr6kVdVWmpw7m25DNlNnSeFSe+MYL{>tJIs6FidCC?!t1<}cYL&$ ziQ0%d3?#{#13hBK{BggEZ+GzBOGy`6w>oZKedD2f9HY4eB$id}JXV4ifeV>$2e%Vf zQd}nUF1ocC74mLp%h}d!rX|rLz&Lk%WQ3 z;)hSy zDK^Y^GsNQAjvP9PJ+8$Qx35|{&s0z^`GeO2ddSQ-gn+mWu z+g~t$5!cMmK~L=2Fyo4eg;#WJm95Oz@bnaR3%$3qEUf<=qUf&X>rfqbeOn16o72$P z0A>SR@M$+vNXAR34r|PiEa|I`wfM9M<+n$e`irW|L;CbXWCLAk%MdW6J%MO^F(`9k=8L&j z6RbSt-Wg0Og2nfcCgYY8CfdxcwcC!&<*6GbtrqGE_m-8%w=zg4q@IdAm96XSp3;ep zFaL7W;5YkN^3HrKdf5zNC$V!?c7ronuQ=Lxl9!&0EFsW73Ag zII`Y9O6{GEsp}_Jtl!5~wS=HKosFtcB#s^|JV%Ck}(Tg%3}WhRNx1*YG3_5zx*y^ z5gjK(Q-e%SlaNv}F6sR&vxVf>lijf%2*IQDB;ykDsvXX$7@>lC{dY@Qr3{J^s-DJ@D-#=7(3^wxgL4SF+XQ-#`?E>!GB8Ifa6P)OPuMM3Zo(>Im| zX3RL3<&%20*&f1K(CBo8^SC?)I*CJ&&+K-if_iiNL;x?!gYO!WPv*PL)T}V`x97{QxW!>FbHGf3#IB_ER)~XlN)|%>&Fo<3$ zqTK#Iu06ykxuXPUlPY|=Jf7p#vA%84YU~$Sd=#N&mWDHUe+4+ul6U>}&^M?HJpnN7 zdD{sLj+pEfRTlh(Z@(2nmZ3j3Kq^20oSE2VU{K3ZQ*OK4EHVK%n{1YE6STW$CsBZ*4iB&{R?Dv`9PSu3kUy$f$G7B;C~CJQWuYjdVFI( z9f+wktC1YdtXksE0(oS*&^MHtQ1?MfUSr?4o20)MebR)n$}&7*p8c=z)Nk3Kh9P#> zxOU|oqxEwAt}pQ;<4yvZBC~yt*f?81@LU-0(IUC6^rpKEG@iz>7o11pDL+bRy({O= zXZhvWF~U{OFA5jVap%2`7Ovl6tDl!9H_nbiL32MLQA_x4;U>I8Lo{Ao1@`O!iqu#c`( zty*|Ag(?g)?zveqkvWPXf0t(+z0Lfj@h8Ihi~%5=b!g%~P*`2Insa9>-xSfRj9d)z z*1;5Lp`S_b{~S5|tq-5av(Tt}o?I_ZS7!xUkhedfwtErlW{hNfx;TPV*956=eB?JO7c+No{SGQZO zzgf4c%7+5z)dAmim{1^+E$)LtZ8pRrG}c&Rlig%CSL}#67@z)w*H&jMx74WAO-(2g z0(gh?1nl>8WE$*hnxpfo?)ufP{D)B;ePa`%3gCSALJTN$|Iw3)?(DguM2 zyQuToRd_Uc#Gi5JB^fD0c~73oavP}`qu3J%d89FLcWC><52E_0{meR!so^QGVbB%@ z0OL1BxsdmW`oe|xWNU@j8>QBoNt62`${CBVTaS-?I{YjG>@|xW8s7Sh6LVO5^D;5V zdOx=$j%}1zRKb0}?Wx%M<`0zGu4MZaALydTQMX87+dSU~ynO2jCv*q(q*J=E=^w)w zg%_QMv!32~TPHm*=s+6W7~hZto=C$%@57BK--jk%rb>hT^N;T;k9v|NkN3XtLbYuD zmBZ=>EwD+&H3wZ#)PrayG;sj?d(eN0_bv!|hkl2gv-Zq%`ZT_vny1ptfS?HI?=(=T z5e-c#u~$<)Kd>9(lTod_i((+_xaHO9c0GBhGaX={s>pqTB?c?rS3o= z*QzIFH#JJp|ET@{`FG1jpdBB2Y(-o?-qsuy?haKPCEWW)FGY$* zjpx?7kB@`9eo1cdaG!s|&nTrA^w>9MC9Nw^Zm4C2dkyYtvi-u3W@Fv)0(ci8zhG=~ zfa=Gl1~&|81BF9cVZnko>X}lSg-=75nvzG?O9y;Y#XXMvY$K%k&Z9H!OyZ+uhC)oa z-h&7;A^nW5G&*}7xAEMgHesMI&BKUVPFJj`YlqLMG92a>x)JMMM<;2#}>J=2M22wIzecK#W9`4VOPo7$PaRIK~2I* zM(!Q4fz{nAN@G@JlYh2Jm(-Ps~!73N~#KlB#~S zqy^-V5~p`<|8w;1imc`@A-e3m%FFo0u~3myrIctd_t9uEk#2j|tHK}eJ!(0iQV!o* zyGT?7zy(+0S5RN$n}khe+-PzLO6Vyflq5Nhh$LdS@a;+8vjGnzb#WB#u#Q?$*W#~1k z!#|ihD)7SlaFZK_HzAhB9E~Qt9L|C7rcauP09X%-2 zsXDxmI8K1$vn))aqZl49H0Nj2syL(6T=}E)det@A)RSAA31}S>LvOhdK)NOuslftoRa{k_U^Jz#APRM zt!NN!HhKU$|9cBf9BlPi%m7^-?VeA}&QT%dK6l!2q#4-`$E-Hvd^V@f<&WR_oj>~he*0&SZIA8q-S_kO zem>vN=k7p_hMK5CISQ4&;@}48Jmk<^)G6Oe7rxOvFUIBYB!?1PJ?Y%n zCHg*K@G;2u1|-$EmwoZ7p!C|Yb-gaU4J1Pw7b8S&e&2>{TzLas%gS$cT6U@OX-Pl_ zaMM-Pb^Rt0N*Zd`rA9anDSH!|E5_SXy$`?n+E3LzVH&miGqEaLsD-}c12<`TO>3ve z#nCgQkvua;H zO}`kcTJxJg`2jrn!?z5D!!7AyS+-J}f$@8z$$&ypnK9q%23g#6Z{x03l2Lzt@zDdV z5A$tx7(RXbK-N5b2*Qb+leG+S^F~iS&q24S>4}?iE!HrP9Hrq175hRSlICK#yWIc4 zcKTJ>AQ|7Zjdqe_@1y6zdW1fvir7{OYx$jdeC9}kLQgxV?+Lm^!qoZ{QfD<#S%Je} zyMOj{4jiRu6DSrsvj}=|bBwLHX_N=Hd2>(HJz~hWM(|W|O&>)KomW~HdSG<-y=dg) z)TaF6+N?P_oRO$~-PMA$0+`)?9_Q3&sUh#ttTI6>=7sQ7To^-n&`Y-b4;Knj>PImV z^(v5WW4gfMMZ|5|@xVoQI=DvU2a>yd5W>=vEPudNk!s@qjNO8V7RY&iUiK*1ctjII zt>x69mD^6DGqUMg1dWuCH?aV5zTbxgSu)ge(pel)+8}TIj+23l)t5tTR-`6vGifk5 z5RyEP=Zk^OcNofQ3gA(RCp1Rgz`zz|>4Elbvrs*UV=>It7Jk?7=W<^SamH8{BaOY6 zby4m2j$BkQf29Wl5s`gS=a!3TdNM&an@YE0a3Q{CNfutN-sUAN?$XpLXDM9N0Gs=0 zWiI~z%VNak9@Yh5JiB4EjC%t?ksXRM@4fZa`okCc{Lf9cY~-|S6CQTsCcn6|qTg0n5h>UKGz3)FkE8V6xBC#GpdgIU?h*|C zTHy9==dl_;bj^No(=vbs7xKsf8?JQ=aH;+=0O8hpc(>qA?*9yd`tNbA5UPn*s?33w zPhgJrN21N5sXWxrJrbLuJ#Ce!R3aOP{^9aOb}$85lP6w|nerRPn@_x(Ty6NifcRal zKGjoJFXZ8A&g1S0^HF}q?ZV2Q02dPL^9ZN#@qX)NHd$J_!P>05AYv^VW*Laa@LTX> zaXu(;AFK1<^%ky&u}1@EBV`}W*;v^6)TNwD6p%{}X|XfZ){MR~U6gNI*(`mdzazsF1qB$U6C06$cHh&V|8 zV{HZeK6v~|Zu!V>ow9nvqOAGPu+#VporEnvjpYoFN+$m`#1b&pF!>Ip@chYb?W< z#OUkZgY4bicdf6>YO}hj)un50SR#BTb_O%QgJCgmj1=3+=a%D>i&?X3m_uaRb@jQk zxS3;S63-5fSW!FX8`+I#1f8U&0hWW^P7N;k=v$wVGP=xnq(WXAN_QfXLKMr}pBxO- zmJv<3?ZESGu!b#Mg)E#=yoI_#v3O&(n{Udq@Q@nf)}Njzp16Fvs=s$0pI=~UHXd;7 z_(|f3JT3^z>3n-;68^?lZEcjlTvzvs&lp0z8-1vmhddtUeXQV6W63am>0JO(62&ce zexaFaRmhWO9OsTo@jJX#m$9Z_?!yj@3Di?TH{2lmUAu+q>PJRKI9kBHXKvdW*pct_ z6#FlaE{coUZ3vczUbw*b4?8Kw<9Py42+fsW3=mW zQr99U;Q1TOjNUI_!cs!tE-o4~itb?N<7xuZGqLpYxZD_R_Ndw(GofkG+#WPe3Q4@_ z5GD(Ww_@l!-c89Ew3{V5ov7hn;H4yE;WPJg9L$D>pDicOI)z`WM2N;VGi)qbmxhcP z1b;V5@omG`)z+%lGsQA1VBs%hy32ztHOs+4WJ~DBj8D@m)vc~_PnfrH3_|-;CbO9C zC!JazEq8F-H{n&d8cXceoO=0_Gu`Ou=uN^t1DhN#ZpcYh96h@Qo6)6zVhawG*&o+) zZ=cnxCr)RCTaI|0&?)VO4}61+*PL)244ZCRiH=+GbyJ}x#kC(?(CECeZFYW?N%?x3 z)lDjAm8lz)>0K>J<1cwpmalXd+BJJ&`5n#@rlF1Gg2hE;L8Im};4BbNO8rD=lMeHF zP)^K94&NWh%&oP*DebKIu5S8S6x4u}F)!?az2vXYcV(lj_I#f_B-bNqqhw2Mec2zn z8=fk?YXioq5sso+>Bq-6c|y*c_&j-emoT!x4K97_5W{|?^WtqL>(f{zq}#csppu_0 z%eG zG1)VV8eLA%sn;v#<&lj(`zxVi{7JmV5q?dHSQZHEXt~~FJ4RUYXePFfd8po~=Z&6R zTwTE+H^|dmEy6mL=jcIg+VK)xY+9pEUu;jIzUA?WV^M_bOM1A_q}V_&4mS=3!*>)_ z#!v)j$5p^;bCJ@m^+Tvae~eRX%EpU!+KdOUn$+IWC%t;qAx?3Y2}%m1aSG%%EcrE9 z4vlAzr<5JzkW3^x3n@KA3My>Rv0k5~)sZhYWO4ar2e6(C9%KW41wK*5)BE1A>pj*g z!ad@S3KZgK(`h`K^`Mx!w1b4r>X`cFn>T#7g^~N13wf;`s;&1a^rbp~Z?igHMOGVq z(6U%?e47R9!(Gy!Medv~x+pANxCJwdC;&m3(k5vLM=1!Y34hX4Qo diff --git a/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png b/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png new file mode 100644 index 0000000000000000000000000000000000000000..eb7c99fd655ab5e0b5691f41c066a080eb7b447c GIT binary patch literal 50232 zcmZ^LXIK+m*KQOMX@Y`)A_N{3u~0>jE}$a4Nv{!6kc1*lkN{Fuq$o<2-g{`F6AU6s zm6|~4NDI9rl#t|1eBSRp-;ZZK zcG}O0PPf9RAW(a+=6%)2uWZ-IQK?r=;QQo#uxxPV2TeA*Ga?#R9UpOpLQc=5-j+Gt z|HkEXZ*HaX+=1kxYhKPWTQubBw`We$Z+}RByUX#Mle2eyeMPg!|3zBy=&V)2Xm@k$hixe)0Z<8zq+$=n8NI_XcZE^Ur5 zQ;&@2*KShHN^@4cThpf=7XRmB7Dqf98Q)KhIyz|CFOr=;FeMK zWD2TV{^)s&PQ{S7mrYCblnj*l->35oJ`RT!*d0YTu5LJbkE~8-+7gP*ilW3CAGHu* zvPI&GM3ip*;_6BUL8?U$GvG~-@wb|Mh!y_#c8SkLOt{vaKXt-^AENd* z?qrP@P&32}gUGRoEaYn2e9WQ0KW|H!tNIIG+Kq7vyE|^fx2IpL!sm-PA}AK(@%WRV z?}+uU{s+OVg)b%#t*dl{3iz9T)p*Z4jPN`tKY4K!rKuEPS$=JoWNoDXZuJCU!WQ_*#(!)9AHpp5_Q)&i>;%`;)8&kI z@5b_}hi)afxbf3c1!g1B-~9gE35R+7UL-J1oB^gC2%#|XdYbH5Pn;=GHWtIEkQN&C z>RCaV?GCx|VNYyy1~+a79aQ_{6?h$I5~gN_2fPD*=huzE9>M663li7-hUKh@s>y7a z;+aO0pC0Y~`@9b@LwVSnq@AyGK1ZH++`j1C-jZI^Vtf3&awTA}jVk1dGrnN8;NUxD za{7fXBsG6E0(Y_vtILCA^hM>NB;W>y?}L6ht|t&ugD$3%dN*~Pd{#*+nTz{2gFN*s z-ihaI zKbJ3kGy)asa3X)jc4>aIC_W0mP@77Bws&vRnYx$ud}Mk8S!he*4mwu@9$NL{vq@AK z7vNBSoWd5Q*52U3>nCMFpda~S zPm7g}!!W1Tg;|>-q(mWBgk*Qw0Lq|UKpZ6+S)qK``<4^A0y`Ky1wx$m`%u$tz8{P8 zesx?9-S5kj5?S8uUa$+?xlQoRurkD0`?gFb-ut^`idZu#?KyU~@VhtFD`H~5jE?l9 z%a-kYa4Hjyn{@)vs3~>FWl?&UKnvx*Exm>^D))0>KD?F=ID1Y7{8-TNC|T(EgUMiq zO_fgwT$mc1WW#%fnwepj@txeTUAZJ50rRlid?la3up(ByYP{A6-V-Q*za zakt8ZU+reyPUz$2=;JBfaed{mjvZxSRE`2dNnnIIJjV4DTyi zjUC;StH4o6VK6}(J4@H3{+CKt1*lw&aeBEggSl$sV2|#^DH+q9F^G><7 z>TP}NUf!D>*sQ=sii7SwU@#vtI)nSIJGqqYo<{C5fakM+#+-tqkz3V|8q+F-JiL#W z(wF`3V}t%_8kOAf+sZ)6hr_%v_ZObuE-=HI$|iVG8hf7KUXe5^)5iXA$OYN!x<9vS z7gyMi%A+K@5mXLJCmY)YwqiRo3}yLWS@~$BQbz~c>stHcTJs*e<@|L{4y8M6#_0k4 z%->8d`7?IyDl)cJ&D>hmCy~l1EkaP-JoEV%Dk3LAENGdY^rQX=*EU+OG2`~C5NGY;K%g(b zD9^0+Dx{8r{-CMDLqQGKrODcH!sPSJeB>Ro;k-l}u3Eou^xT}?VrF%%R64wseung3 zzkjkV_vi@jUn37PeA2oY!RUP~x27`urGlRLl}KWmW?qRbLGFLr3Upy9E`h(t$y-b_ zj)O8UKzN_U9?JM6_pyVqmHZ}i{4b>NXYnD z+;1MJm%Ex;Y&#NJMYi9Vv)8pXB?@B2+)BCwkfgap{@9S3J?8V$Ew@T@>NHnDZRWKR zu%=%s{B~w;{=3l&iT^A&VY~{7v_PXoNMj{*fho{sT7d1Fp3Mi`F>MfUl7--HoXQ3# zLBh8a5TYc8WPa{74#})vT$>IhI-b+fH|~AXJfSB>)#$L-6K29OEj&-UKc)u{O6ecV zmo$TSO*4F1M3iOOsbRUq`cKwN8T6QVD(hhL;cIPvpenebK<*VJ=I7yG>Q;Ky;>gWU?%7YPt2x!8+7m)oR7hD|5?8{&Z*?95sL8Gcku~W6zF3iujA_QfN({piJQW%5C`zw-e$Z?lc%w_U(i&%Uj}^ecQ=R?uQM zJ1CPM@F$?QQ)tBc0nGEo26BGmi|Alg+KrZWq?oT$GN?zTSXi;~*=QZTzE9H)#ohJC zxG{}gw8q>gSip?QX~;-Un4|eJ{s41m^V&7lKACN-xXp0G#d!JzsO|1KB!G5wGJ#ufkuiZUHxlvbB=BQ5|L$mf4j5%cp6R$fSC@Gb z6zq|-jwoA{)D&m_Hjb7_u8&9ci>~q#}?yJtvbS)|tRPj1H+QjbZdY!Wn2g zrjBNfRI9b5ecto%Q!kt!Yt&vHl>c`0Qrbt9m%4Jt+iCW#Jbv?rxzZfc;Q?-d0Qi02v_c^XBTrxt=UC=Ew*iKuJ} zMq#Q*to2pJp@7en-%=5#l~{|P5#79_`n5(hKa%xTad7V`w^Wu30ZyC@r^PtOM8ADx zY(#^4h!tY&dvBh;J~OADB3cXP_ENN-Pd~h*MJ-R}5e|o#4oSUZD=JmsFJbHtkvEJRRe+Y*YOG)N&8v-s z`6(FDCgTA+a^#>Rh@uK#{jTXvL!gu+H&+wii9-)UU+AWcZ|w@3vAjFZ_t#atkgwP~ zoP*_~6W@i!i8gPqx=z=)rfOzAfw5=NCAVwH6O>rR9eO>j>Vi&w5>92oi@YBlTHU+22?1^0{_Q zS4*YrnhlR*P(_qQ_@)_Y->$gapUgXKcf4R%to=TNice$Eo$={@%~s@6(PRt%;{Z0B zCeNbYuiN`$nYZx&$idS-jYfWZv>$iB`kSix^yv#GBw{VQ`F%KI-3w)~|K`DYo~Puq zp-ITNoEi0s^3A5)){ACb#SMs@B3ChIV{-q}tu!<6r7ySx`zy;iMe!T5 z=59EhA3TH{$}5>Z?;@AUE%Vdegrq7tHwAIL7dU8ytxmTg)?IO>F+}e4Hc9+*E47C* zNC}(w?yN~sX`!*JmR;MRw35#6y@L9bT&mpONI1X2(K^=e6|i1qH0FaQT**tlqc2FF3h*poGUc6xusrTCb*EeSrf)yd5lI`oAN%+o#%QBNu4%p z+f--54L0hL0_Wu|3O^B1f=_ok#WP3&XS2y~=VNg-4mvT$F&qdmG77CmSsd*3(Ec&C#`<3f_i=8F5gd2qXx`{&N{cw#D)bSscVR1@7xC1Gn8y4lsE%B85+neQ+N)2~i z&(b}aqxVy-J4df<_*!>X3lp`cC@qDS5Mb7Ofnd%-(?sGd=YZqxU*I$R#KU5mn^g5ZvElzUNLq(&o8z?S!Ue@z1`XUX*y4>#98kc*-MgM|3bPc&DXxH=)EiU4>XnQZ+wZ{W5b0x5_z+arAXd*$N^WR zEd;w5g5>wj7|{X?>DtNOC9O^uQCo;%{!8CdM;0nxr!nG)cK*rCOrm`CEfiL>lGy3j z8S;n6Yz;Xu)8R0>GKdBut(@1kR2!luQYwy#rVB0=H$0FiN&yL;p{rsSe5@erJ^SG5 zCH?cCM*3~Ee}1aTS%7uGk^-eKUlUDzodq5Ul>e$<>u{?McF`r@g&g5vJhb_hc>*6p z4V#ZmT$sOJw>hETOQ?0a$uc={n`4LCV?c)Ycq)Xt2=hsT!zoGq} zRF6Zhp%QOqhi_we09MBS_hLhE@9C#}M>Yp_87;Z|)N$G2-pxZ7@3DnMUXpmr@I=RA zD3V#c`S&T<{scACpBN4}js}$AAIy)QOV-u_$wAFn5_>nlGgWQQ*f>*Alo-7md zYO>&shCFPqKPlYfbmo5c_7y=382bovGnY@)?WSu^S%&50L?rnwQD{$y)hngb9O^=0 z5AsAknT*liN0iAKxCFTA+@-uqmf)X6Js}U|!!a+WPUOyw`huf+Ok?~moLdc0FF__45| z6gP6Az2^5*;O%P8dvq3ie?^nz);&3M4>6FY6+gr24EMkths4oshewP5B==8kI;@HC z+3RvRv)p@R_t-#$lPO%S=sjgpgxm+cJ?oIGX<*9R75*)S4IRQ>+AbUM0k3m|q23e)6P`B&YqN1_>~@~13+Pb(hb%}|XKiDxJG_+S~wdrQN+1%cVN6cn`r zdp19<>7Fb%db`72h`Z;{?1Sdk7j@;Xjnsqke8sKGwQzrg&C55obv9Sl9US1-y_{y$ zJ{VZNijV%~!VDg~mlqc)Psq}&uW*MeNBZ#|28PUZIIWu9WAYzz7m1G*T&&aWuYMn3 ztFx-n3jk-gQd37?xH4Y$XVTaIQTsiOhR zMXjm2D)RolwV0y*QpHjVK_xKDcGj&1f`=C7ArY)i+PVlIXoJ%s!`>Tx zxT%fjq&FTO1%6E3`cAcjdl0(uGre_%w0xg&)|6kV7yzwi_%ABQ78Cr~k3g>+uTJQC~^l`SLIKWP(7T3+v`pm4-DIF-^VzpY7pn=VybiI{hX5Jlb9@KCV7^wrWPz1aQ9 z@bP?vQK4z6)g=!nJh=MnkD0rHUlTG}1)w-`N;^{S4)wQOUFi)n8ae->(&@QJ2^z@} zRhuR7w?pxt%NTLbCJI!j^t-ZSlsR##k&VyT!lE0pg`^ZzMxkt1q$rH%(y;$CJ4$TM ziWIfrLHc`2T8r*OYIo2r2BlS;sf%hG|# zQIB_XrZ1$}jy|T9c7N#ik&PxjsiAmXw+2J_-n?%KxcOBCN{KPEpd4a#Y+i`M^(vpA zzKqnwIXtfohbc=igRe!m%WxOcpGdaIIOSTaicP+72Ry07jZxW;Yi$y1_7q)V^FeNXxfAZ-v!ld}5^a4NM|(>>_6O%*w_9Kk`DyDN7Rpu5j2B?1j_MfXe5fk2O@K3f2xk|8FCCVZL>> zlOy>w*z*BfK|N5xcT|g)h^uVp`geZw1!jqzi)7(_3FoxjCvESKvv7m~i?)N?VNq4iV^9V_J3zsprCGP@$SapFJ2 ziaR@=PWl!*!sjbmbhEr!YjW!cy*F--VT8eJ#%Bg5?5?ocygPh+tV|eaQWgZju-v{Q zgX$JxIE~m>{jMGPhnZbxQY4_7^rsJ*Hlzgg{5C$(oPzPqW+m_xJ?@ez39dl0|7%Y5 z0pA+$h7w(FAW!8<_C!(Aj{1D<4EB|`eDPf+tJwI7j2UOT9 zkEOh~7WWG)s6!r=e4w~7M32gpU+0j=DTO|(V8$}~R z=p*#b!@*~|O_7-FAaX@*>t#1HbIbnKG|TDb(WzTr~<&e29309*`g zg4#nGNfiBJ4&MHf+W=ax6a|3*zxlUsj?I7xlHRf!HJ%a|hIXr6<}SYfCe5S&N zF2aPpCtftKmW@zq5Wii{(*5USUD;gREKA5~y6o2~-Yp;O^`1Q?KiT)E414IBhfk0|uFj}0>`c7;8=d*Sh zNCTr&u7U1Z`x^wf=Dr~&(ZW%^_=)S;Rq7AVn|v*cZJtIMI!TUcZuGejq8`4 z_UFbDd+TEG;_APHa&)K1@;9BBf%n)eFYiR!;m|L>E$*qo3(Dk0%g~xKm;>lRou_~U z*4(gt8GG}6Y@TTkg=>$`*0LU42X7ENo47Y=8)=Hs!RBCaZPzA+r~4T02e!BgF@5N2 z>ihbXWm_l#zCv#D7(Jc*P7sYOC%w$=d=z;Wr>9I6$Co#4ngHxaTI1SI_sD8_y;{rS z9DcxXeJ*Oh80C*qB^+SVAFIGuFn=Q)KqUrq{!h@R&5o)ugy@-&E=SGak!w$-PQ*?l zl@Z;fxs#Rlb}J4gMDK}b{2_VkxhL%JAf+U_&>dc>C%jqOpaK_1tLKYjJdfltvI1-8 zk;#F&cg1Pcq-Ep2viBrGwh-`4X1gR}5fIQGs;0K)=rX`c><$|n=wg6k-|x=b*D~&K zUjPbfgDBp->0}rHC0}YvB%EpI;hp(O_^$9Jd!`3}9{&seqT!}COV(UlXPBz4?@KvD zrz^Fo$1S&QRCvw&pN|P_-T&sr2O}&O##{a}-7mBBV?~f1PnTB3(W*#KfNBj;Ml5@Pjrdso%`}ny7IXb& zj;fl|+GswA&Ean`tnYYYe(PbS=IqDB@JHq86RtdScf!9Yup=89x?&0}#3ta^dXh2H zb`Q^sXhE!eO3Cv`-SyE|Th$q6vdtc<@Z)qxLSE28apO`e!!Qlwiq+4Y5o|77Ju-eGuQG-J}W(aLn?o8Bn5B+q-gjDTq)fzNw(FEEp7BaY3X zCCqI-FXbju?m}Id&-)5Mhk{CUr#pzFbh0#OSunL-9!e`>Nyf&s0@33gj`n;Ik{iPN z&~+D*dO1M%{6_tZ)|vRK%v;Q9(0YzhN!%%KfCWcn_@p9iFXbO9pH}_E?v*U}QLVD~ z6B;RUTmnHdFHJ?~i0-|Fb?z>9a*JrJ|3F`FSI%Oj^^)t@>x8%@|G-%FE+>ydYSsaHoCXw9wxv~f2+3nrArU%?`;OjFG z92968XfcfT#Y61F?QjGP8T}9RwZG>2SoO7+okMCZf{Ns?{#usGNlU|g&eXbq44|zK zK)>wm0oQ4F2KNkC8VLi~e%c+Jk%lyK{WYa%bCgUYYbH z-?dA%S8I7{XIv}>WfpI(X7+%eOSetJeD-APW#-0<#hnXD3d|G!`f_$k;?0y#!U`iU zz&@d`(!!SC%=c@kXGc1lNgEetdP)tu_?~S9yT=6P2_CpiMqOEm0X5AZn7M(D~<4M;f$D9AoT%h>E4;~Xxwa#shp zhzwSh-QeI$6^U!qRI>JrL@?sA?Ykl@S5_s0t8uw+tcGl!@wFqJP|o&AH6J%_!(sf% zWnh5%zsCpk(fWc2pOqfnofbWnKoLdabwx{ya3f?O=felSp&jdoGkB*;YL(Kid5KFW z<9-DxMfvEGxF^ZHmQ)MFYFuBReBo;0VE=3>fBT$O=SCAxX!$0t59zsnNPr*`dI~N> zwH)5Brc~0CC3e%49;Oh5;NNSqGAttv5N_D~pbh^{^VN<9#_!hdOX|rS#F!M<*z!B{ zcYIh3?Z|wJdi8y=9uM}%Fp}{7*-sbNZ#wJW#H~rE*RLN&nx1A(?^B^3SJ~)`yWFrr z%iPv~W{Y2xWqlIrStJ}VvZRUeeDPBaYs%Mt&~oFlmCfGvWPTI-H5Hi^K4;BGrV`&Q zj?A5Ttlz0xYHxj6xrFS1eUELH*)$&K2Ugm7VA06&mL?fIj1+;~%T>R`@vvhyX-~Ed z8hqha6uMTr<~$l%z zBDScD7^+9*BSiVtrxGSd_xsD)j#fz=P>x!`QyM-kR-^L#3M$TRStN=3SmJjM{UoHC zzlHIN&+}CA!|gls-RPvube( zz)U&f?pE>gdRNo$2qxSo@*X4Z*dKgj=gf74!ZA=G%8$ti(1R0xxusC?7$YSRR+knc zRk6=P5^g*m5Y(>Q;uDKaaI|k%SPvel(}p z#Qg9~Zr5N!Co+mZL$iUYH15NL9EsI9o1fA-RQKNu6bCJd zaR^+?SB7eztbMdo*WYWy5iS5AC*)%p0sj}Fz|&ylbc=aNgCPa&xi`w9U0=<$@i zR^fup7yq%|QP*Oy&pjt>U&N?oss4FuuI1`iDOkhIqiT{HQ$a{ziVq$?HmCsggyCC1 zd!r?5K0H^T`6fP?&Blu%$7%p|{K)MTOpec9cHph8)^)6bSh;U3?rn3#l!+Z3JX=~D z{Y>fnK`lcne$d){db=V=p$%I2rPcK~By3b%$eqj>B958{lFX+W`L5Dae>R#7-8;3a zdb1q-18|3}vJ8ds`00!{4eov{vu>AaoM}>-9(nS#yWAVz;K}z0R-wexdh)+5Of}5N zLb-t!#&=C0p|ACxl9k8)8ihT;KC8PZ4dwXpSPT6?GM8yTaotM$NMw|8?8rH&y}^kU z_uZi83DRv%T{aCZ1NX4CeabNVE^q|k<-%_wu|8d}yvAtUD&VUhe2Xo^a4zowcOZI7 zeaSH1SWcZ}LVE=4Cdh%P$5_zvvHl>>j@V4-ed=Y{p4}5$1Bvd7{9&_c_XNt0vScYCI8a{PLtcHo? zNIF;N_+$1+!Vt?9;pm!?)-I#M;P`6RcGF-hw;Lb_v%@@7VkI-`>M2AnG_v^drnA|^ zp>=u`B(9m&@n&o^EA9uER&Q}iTNi^;v@iIH#NxT76_(K_CYh6Wpsy(!;?E!{lqX>Z zjkXGh{VMD@y5KZ5V5Hr21&WAYp*O)|7V?FR=&Rg)-vv#1=Dx4%o<}+xg5UehgKlkl zQW3pX-)s>TYK_s8`zg#zf8#<|` zLjaMin;PWw;4E8R5z9Gbp2MfU*jH1Znn=h>cBIKwX5i|j*WRVb-R3J$by;uU$b~FT zzXFVd@>}MUU!=b7qKN@k9`0sS4_5lFuINdjN;$&~VX9eX)wXpD>S970cH`h_JGj}? z;fh-V%$r=pB-aKyaeSyFF?njtG{!|!>GIT@lpebVcztZ4KK=D!m*hJ4;{}=4&3v{I zIpa^n_&BsQ^-NKM@AI_hLcorg|3$RR z`J#y8lZCeqc2>g>B9?a0RE9f#pu`BPL(12k367(wEOEa=&NJw&8WqR8B85M`gjtTi zN$rY*CLzkA-0jpdY9!92F7uqmu>!MJ{6Qfp6_!r%8LSZnVKo7~EEA0o{U1nt4hbO+ zpOQ>xSME5W#f~_20&0N~FJX=eBMu=%QQF;r94Z~oh!bnizmco8Zx;^pORIuq@i{kr zm=^nri$I8`Khx`!2^Q`B@nNd&?h?5s{3=^5OU{~&5!K-8qK zj^#Qfp2B9AjkwC({8fraEA%?=NzHG%mq8c0W{S@<++Ar^AU;@5t$~e$uh(d`3rofo zubLGC%CXw`{a=wbhkT*uas95lNE}**Ux*%oqm}d8&fr8^*^#|~K%*!9tZH0*IOJhx zu(s#7p6FPmM0&N9i9kWt$*y}hCVe|wa|IJVx><1}?}Y0rTtJ$Ta7PGH7T#S)a=Xk5 z^Z7m5xClTR5}g~MdNO!2f|fxLdwxCF{dscPQ&nCANDSy`!-y34`LP{w!DQCV#WHBYn!~P-O}or` zD@o-dy|c0tbNFiFW{`jt8|W`98Vt)1jkUNM?Q5zzKi!)v399UkfQ9e%RT~8?rI?<+ z8(x!EPZs#8`2`aDdtVGb)`cj8Ui(d?XSBSSYw;|D4k)%}e8|*G^eZVdj#HBtO_}$n z5~~x30YO`VF*bj7Ya!miN;Ayu8>qNqNb&tQ)-=>pKq$2c(e-BLduWJ+H|svsz5;a09Xq+ zB1})~4)Q@JLklVgRZ}PlhS!~jx&c+&+OvOKm8jo4TXoNCpEu=y;7?zGaS^2=P;`VY zc#@X10S8izcvn~zC<_DJ3+)`pwAg9#8`@zn@b82ElSL4WZl$*&aKGOO@9H_4DvE_phPzob9U)#&1l>=A z;kK2BV!R+4$ih)#I zVq-)14qz1xI!CLCsSZ51-1s!)#m8g_OWH3!LsC`rW-Ij*4(DG`R+l9G?G8QfDg~PG zw#2%G{y%icz&n$ty%DI>(88QNo4d7vk(jn!6Ko^0|SH(6hCME2e_ zVD3D>-vDbdy`3*{O6r6WdvN`|A|~>hOrVMfc$Xk2D_%Qan!IEWKCD0LbNu$9WVV*` zRCpM{2OMaTIB)n4jogzQTqEp#Rt2QSyZxw`lS2|8R$p6N9naUH;=`Tfp(ILdkkM!n zATC8htU!p#|CToXODwYky|qTnLnZY?opx6b0cDo>FL%UyA?MI~Ru zX|AuDt2pE@y6nXn!40cR0>iPjt~$;JAH`i#Rjg!RG(F~U_hJ+cmmBh2Af1a89XV3= zOx_pb*WULOP75vXya^CNi;{yh`mDutX}F8hV5suYvb;Z`W63>`qT_GvIOvdTnTUw) zd~{+>`hORdb6;bhiOs|tTxl+>V1A?M49Cf|&}AF(2pmur9bml1mmdG=wp{Gr6Q6AD zxDT%t|L}zc>D8Jm|sAS*^9n@IvgP6%+sc;F?NV38(gVJ#^GNYWAO!EfY)bfffJU+Z2XNKby{d2t+ysuW`?}l|%gtbsAxev$)d!*xzK7T0bwnkpoFS}E(iAk1e;;-O=q{jEm#z9gsd3y2Ufh*f0 zkF-$&szcpqdH>X!vmmJKb*Mt172z85eVm6Gf$KO%!FTRpbno?YD;Jps#r}@QG2*Pw z;DX=nG2_twiy^J%3AV!;F#0#qUMHq~48h^2%T5^XQV1^B!`hEbVoRp4Up;#)rrsItRR()l2{uty1w}3ZILw z>#5GG$_06jgNG^UfNaOb4BWI^-Umr7Pc3H%PW$cu9QT>hu4jqd<86-oDm07hybkaT zRV{x>4%DlP_z3ZDjdM?wQ3k%V_pUb-P7fk_QD^rvhWE4LH_cX)K4oURT8z1l=$TA2 zT>5qtA!)VZq50!u8C&cdN4mQO%lexQEX3K+#6n$+YeS1bc?AyXX?9u;)_IdX9M+h` zHuEiwzD{>ZV;hhWv-cWGs;q8%ypJj25ZCU<;Fa)+1f_DDN!o^&Oj8ZaM zdhrz4rz|YCFa`Fa)mdS@qOAJWKjpk+zwjEBl-2k#-v&he5}k1y1Lw!!oUhZbf!?c==SGurf;pF!R}Cz9%oEb zAKcJa+zvaW`aS|GL4R!s%tsqBmf>u~x}0~U9lsMx;%OF~SJKGMpX02M)C~aLfGai8 zGJIzpvmP6WvsL9fq7$iMs`2dh9i-1{Wr**q^N8LJ1)L0XWAS5OsBU> zTLU?_E);hfmt>Ybjxf_*Wdh24E3VSu8wv0;xOW@FQJY;+g6VT@sg2vgYB6&~n5shL%a#e@eLX5iKJqvD?$SN>_WU=H>Ue(h0j#GpsP< z_{MqU2tkhJj^u}Tn^2bnUI>mKdG*AX>zbKdo71j1PM&;6V4I6D{FgS_Tn9B&8Gcnk znMN3XeBYW=*8ML@4yZD3^PXy$fi|I$QlFW|McPGCrA!lOgIiye78~gyRu&a&zIPHz zMePd2;yk+%w1Ga5d}DB#KRoMqUl*os*d_Zz#df2T8w0LaIEmX0#YVdW8Rv>PcdV4 zHr#P?QP)0Enh>QgZcdC6n#^r`btlrf>K5OByE!kNoKDEUiiXei0GtKb zm@5Dl`vkfka9owfiL@|G60YS{@>xOlCtO|kJ%g)}{`tCi!wpGb#eEVOs7s77NnwKm zbPLViA|Ajbr(wv-=X%O6VmAOQ>*Y?l&VAA} z`2C)CYLZWaGFks#7Hp#tX`^648+QCZe;Y=u(i+MKwz~ig1l!P<9Eq~OW&y3q1Fk06 z`2qH{mpi^Pbr{&jIh?ML!GZBzzybTyJm>s@J0{DIt!tKq30uxBYjT9)A-|U1u695Q zsQC;+YP5tDh#Xp6`b0_;-LVe1MoNtUtvPkcJSsD#?)Z+;&HeW^x61@YeAB5xqe_gt9W@&5Xi5|Hhm+ z%D=lZ0OU@Mi|vPQnYM@}OmLlsioNPS*|6bg3T#AB6MRQ1HGz=yq?Q*nc!TV>AC9Y8r3S`>BV;VGS7qvwYFO0Nd=OR~0p_<>bi7-qw0z?Oc z^N6_GzIcfL>-`fNEVv0}wOIysFmZA_iEP&C4sYlWaW#gbF6Ru*b1ML^_M&g1@nM7N zZbr~x|1Dn%b*`W1+fwc__~sBhdQIqH&182hmkGCK^J3NpY9UuyxVXq=5&%w}blnbn zJytCp0aLd2yY-cWbg_yEevxzUN2K2NQcb==sp(7WVwq3ohQG~6w-a2_DT3N`4d9mgxL77#IO)}`_8;VmEs24@a8dL7BxUv4w!wJ(2dKd~F zC&C?MTUufjNW2wm6M`KqYzxR=(i%F|4VsC`OdluzwZQhK^FOkoDLcZ4ICtZkn{j4-AAd?QkMjme?hr7 zMZoAus`u-XgW|SUckKSOf^B!AWN<$)u|XdB>m=CKXfUvIyY21RjNtKvL16wbHTo5_ z$`o=~E(x)tfSb^^>&eNH3C01Mcz$*M%qyN8u=vt=d|h#o8*yGm$cGL01i8L-+gZ7_ zIDK6`kx~;Y$yAvOAXp*Mp}}3r9tc-{ z1Gy2FcX$c`Y}1r@N#S<7Y?*AQwUPKltUUESQa7+N+3s`4&JJDdVTbjwcAy`?k{@T? z-|!s|9J)(er1yG)T17r*Kt3^(ZasdB!S(;soACu78g1@^unQ`|Ir_3q~?_#ZT;!|6d*F#U1hfjI9tY*>`A!Hb(P2~Z$blR!R6dfVLyuzy2)-r zy&DG1bjWpzj1h#^@vDHng~9~}tAt<1aqZ`bLaRxiF`c;a{x5@vvz^0vD(em()NZxI zHwfLiy9MehI!y!H+XT~U<%Q0wxUV=vm3Ff4EAYGZYx7Jvx3B8XLHT+E35O?RBV4#a zR)E73dLCp^OF1uv`W|9eNKbkrWW9)ci8?-(QN;zO9;io63MU+AizeiOwQ^w!L8@xM zX}f+uhB1yk=WV&gEI(-9PA_B_XOFwS3XA}a5M}F|LyhG}o&8g+p*fj&{nRQcVG@iy z>Eyi3MlxCM4&?*r(BPs z&VRl2$n!+1T?U!8F3_>F9T~IxnEoN!+V@NyOLfITQr@@E;OTbC?Fz1d4beM94SBnl z3rjD`uo^sy?{+_lYAaLma7RIN1AQKiVTS6ovenL^;R58oVPXRdF1EjHZ=z_*ZNR!| zUcg2jg!K=DwLNB1{+@tJatRzb2~^FpZb5O2gFMJ+gUbi@nA;GmZdQ5_k_EymkvAA5 zS&~bA0!Tk3fD!v}11U@P=7)Lk%$qcz!mCqxK=cWrZH!a+_si?EV8%uB6AA!51kAIx za^MS-nl<%1B4ApJR)}j_YoY!73409Cx0A}|<7*E{fEYqeLF_FyVJnM{ zyXFfVnC9IvIZRzW$DdU{^gxGRSoK~89u2z3ik4{`I)R=QQnUXZ&m-?;Ue~K5l5i?WUZ+Hj{(74e+Z`PN z%xwc5^%nCX<<$`Olb@aed&$BUY6VN!=g!hflYl)pRI#U>90h*+4A^?k!Z!9QA%;|i zCTQy7M+Kp?X;2XI+0lhb-Oq|IvY~)V%yNPnc2I-<5+<;)**9JMuAY_p9-A zTl^-j9jab!GwJV3;p{9ab|Af=dr+ryS0x)Ryz3e|P`jfjSF?j~R#Da`ozk=7M2?)! zhe2PD9PI7~r!P6hWXrMqaG2u$Xj0KgB6{LyihzJ50@5Y4Z?3TSKKtx*?j84>@4nwV zzA;Yz(GjxNnrp7P=A6&-E6?(@``l@{Y)9erywzEI9-pUVbAl~>>|i?2P3BXx~F&b6Pf_k*^h!46Brdm*UONdaWS-WRO| zd3(Y~J{q6T+M_&T%5kipZZ1Fao)PCSb9oZ;$a{qwrx`eDVR1I!V+VK{=zeg-$!4HJ zi8ViY3hc{Tf=KofkzVJ!#`+bvidWuB4ao40g-dK#Mu>*Ylr&AeI6NhOr&W1py?pB# z$!glpCT&e#Y9y2V$;Ud-;)HTVEFrXwb8P5E&{hl~yy@Ox>vY{{F+|Qn_kcz^7<@Tw zR@V1a$RQe?(jMf$;a^s3}Kd&9lmG@}=6SG%x%~;Ya zC67#D&i)XWBa zDoqws-Ls{Q>y~75a!v;RbkgI5sw*4L1lZgaxyXbFxTu_!@d75pbY2-607g&I3~x`E z-RDaXGJWiSmY%w-`e!{Iso8BJbVI=A$+ehAK|Bg-=1~z9CMM=xZm7!#HF0d10-EC6 z4ZjvA4j(wRIz3fP_ei-nGQC}Fiym+6q5jHkuqh#jChT%z+^E)4t=clz$C!+X9E{^b zxh914>dqct#0*Dwi-`pH_(MJ$qaPJR4*h;hE=3a%^URzNL?F)-=w`z5thbr5KAM6U zxKP7mw|nM?9g`0-xT8!wJ6;_#f6wzsG$Odzmb2-ME#2WX?3{kT)!M68gBz^&P)qKK z9PP_t2y-QTSbW=AjOg}N%`aTCCq%b@3Wk+2POAbuv^9Q1>**k>kCvdaAsopkrPz7t zRg*2TXBK~XVtFnzH4D1=jkmDP1NmeYgnY~?yltOqTD7<14&1Q~>A>y*4r zj3or}s0@mW&%g8Snn+x0URsKNV~-5TAhd+_<6=hOwa%WN>pJ&;?y83YD5HZf`$cyaPQ?cR;gcJ}fjyN@wW@YS9lR zXZqY&zF5{t#6<-hXboY{I2)Ej5E>Vrx|)i$&B~7sd6ho!P-zBQ-{<%v~YQ8nG;}*cdfj#q(585 zRn;I8_To2%#QpFlS<--;4#I9Wu!e?7l~)S{io_k?c5bZ3OU5d4l#!JlWC^E#=AJ)i z1D!IH>A#4O{nh`uh@8S-&Zk%*S%Bb|%4Vejw|y6NZ+*{7a9)uni;3&=>gAH=p67%U zJ>s}J^{;>>2^^)sx+-5_wzJ^Z9cpc+Yvwzyyb|$vUiaY|Ru?cBm6H6n*X>5nZK!?C zoOg=GKts*{60{6IF(uC?cYm7p%bvr|M-v}&Z$B}t-D)H@RdC&TXl6w10MwVXE1xv{ z)Q13Z3s60ufMqDELxJET_p7*iqVODrqRKu}nc`rynL_qw9Ad0(-S04+CoV7~3z&jhJg zJxB9}@r2_sy-VI%z2ep7;&f?}l4%id)O(WwiWRU>}4joh1e;>m4rTH1lP$PzpU8?I^ny0qv(`5>`pZC zebW-LMR-Sh*Y&CiBGXtujutRJ40ud*&amzmjD~6TtwJgU8BG4Z_zShs6s4dm{jM$=KTrtUL`dnn;d3^sx)nog zAhy(bnr+hu}u~ly=!0%Zo zdM}_QBs}JdpAFsAxBC!$gB&3sI21|18nYZ!hLstXdLWM@qFy>gJL-+hO#XtUow|Su zyK?ezRad^H@vxhORRP>+ZNlNkY!8x9+RA48rS}-d_WFJYzT_Pix4z-ET>HWrbrbjX zNurycajL0{^UC*Z_k|A+d(7%cDc+m*8o9~XWqZg1MS$|AP+34}6%tp}M{w6^opEG$o`H*Rwql^0FGmY4;Q`0#TmlWzV-jDWFf#mN{3cGBFfQbUq! z<&t3MIYigeslZ|sYMUfv@AfouqT{2^gMUX_Y+)To-3vPaltu7BpG{V#q+@&7I?xXEL9n(m0n)v95U z_x7}J4lojKJ0y4N4;hzpwKKxaMS(L+%zR^zTZ1!j5cxDs@n?c3Rio7$*?UREckRFz z1IVXV79Pg*#BPt}0SgYUJAI~pk0CLU>i4M~MRs790$L(34viwfz?$K_zruDc4aR#T zw;|O5ZX^TaN0ch0>JC3Y-fPc9_|a}M<#@+R_C?Ld>H|o{4d;1o%<**Jm6XTjqBpwy zq6=mmFf|di3%1$2+qN|UN}e4n!soY2Jq?MYX@!-G z7bwCpaqLVh@gp*HDgu+dJ$~GoGA!kLsnn7BQ4`{-{vN0i`2Cy7LM;`{m<+-b zg4(GqnAQ$8jc?tyBx=a@%BUjwpDHa^nn$-{^#LoKis}pCpvY@T`KZhvXuJc>fc#q? zgCHwVa5NYRyz+Pppf#DQ|0KvQ6XR>J(cQ`5FWbnf&-Bw<8a+xm9h9J4FKF z`te%o_q2BE8IIkIycH0hP|=5>!^bJ65mX|w5Hu&!0DC43{sS)X3mS#L@lY0n(QO81 zx^#4OcD5$^;?Eh%MQt7$wS>!+S%Mtu9eny^qO-Rer=bK`>Q3|`pOYcp$m>LHm%;7h zZf%0)ldla26*Gl^3*iet5d)&Z9u;y+RqUnauvK z_+uMqjfq?SiBQdX)!ha|!4sgVxc+(~Rpza$k4ueNuC}GcA8@r>d19!qAF)s5sOfPR z-nKs8xI4lFeJ^wA2&$(ZB!(+AB&$glv*H}iPje-Rr{3l`RVA-@=pOl1g45@qhmHHa z2?m)!mH5a&3UB4*WjyT6*mnG(?A<*^#(ocXyGFi0xWffbASM_})n=+^kox`$tw3rd z&8^S!Hx6FooL9V@syEE(Nv9s}bLlbHB zO26U0<&Hkjcmh6mxCq%Od)szi*QJ?l-xX=`@W_5C_t6imuBcIWOp<+j5!2)H`?8bf z+HBg!ePM#%L*nOuT z=V%b!rTe#T=VBV&e8VozMF=)74~MAemKCL6EkKlZd5&YIxeZSWcuE0*akKl?y8Ikh z;CUny@0F7{D*A+>EhFKlPBNcKdk)Xvy0i4%Ga5_RCr1w4kV?M*m-nmIR;_cgyVyIKN-zdd}^KghDxmpbgjwLVM$VX8cQdM2z^{~fxctxP9LuLT;L^x-?{r-{G#vFC$@ZYTd!i90My31|-iPn|I;q4~IqG$WA`xdm{okbS0Ucvu!Z5 z#R>Z+max8<-`V?ryD7Q`*wM1Psm~YFAAA7ViK_X$22$sXI_LTyTLUL8n!XvN_HxPR!7g( z))dSZ!b~V=nEwJK+4-RGjL`w{>uv1DolGG_X>%Uc1qSEbs0$-7hxYw9H0#d zh#^8(6N3!4uAglz)BbVsdueyFNdITN&CKlQEf0D^=L9oGaE_0@-*6S)DD|*Ks}VSA zbtpv9X>~GRr2Plxyn;j=4UnpxC$eEOkRfoREzfV}xn43TmA|Q}!RVj34i+Jmy?mTa z-@n{CC~9dF9Vo+FtGt(WAtL8OSm@&fekOuwm#)~zrv?~&&O-umW#S`wz^o6gcJMT8 zf63TWra;wtt)nl?g3rT$bskZV$9$G=p2zXOrX|}B=Hywc%%fY&b2hYkY4BezznMKm zaQ#i{91GZy58`6DLhh?SON&Hyx5186CQW~kW%%FYrgOe{`R)#|4G}Uf{(#YTARJNu zl)>}*Spc_D5&xGNcFa?Fyumz~0OR71yG#lJJm%jB`dx4%(xsOXeLweD^zEAGLeoTN z{ih&ZXjQNL)d7oDH>{de z+j0G;6d95+T2r(Wf_4bC zE~FFsQAJfw>V?`t9;R&LI;2V2s!Iif>*dJ(?~GEfBDy+#-y>h*FMN@Stm*}iUDbf~ z-gTj+Th>`AXS3(;?=i8LIQOd7M1wZp=cgVmc|;sai{G?-Se=ZKvr1H-$k_`+Zl@lX zlo`sre4{@mLRm^~+)HI{j?nab$L?X3=U7G_=aC=Es2&HDPeeI#5_y3*v z422o2++dg_D)yfM4jaFA&kzPh>i@8yq9J7b+6nn&x=zX6-?a!qBMcMIm|^uw)D)b{ zTy+<*1{>f_iLV%NW$U3iG)%-Hm7xXBx@H!=%&;ke%VM7LP%Rd7Q^|MR!Bc2jgJfKxt z0xX|2BqE2A5E*D43lm!m+z?y&XpXd%@OKeAR=|1~EYW%{8+zT6Yu*W;dqa@yb~&EH z!UcNG>c{cr?HlMKQ5$<&f+*ySOJxpx3ji;(-`&QIv@ct^0Ru;l>RUIE7gOLQD`4Va zt=qjh!~r)_0*uwoO%|@F4P`>hFfAj>oq~s@7=dOG%RK#?LCN~o&_HjGE{@4>d0wtL zKy)kU5F}T*J(GldGoovE6dY9#I6eR0F;#EV5+X)nSyK+uvNga`cH$V<#{GD?!YQ0D z*W0yMav@z4dgUD(l`sk4#VUatV5-p15(1>!;1%fpRQVu%BLlM9*m!zR1l{e=mNWJh z;WavD%n`lqmGbT%$1Qfy@Pie)oHtt*wH&6CWbqG7;E- z>Q9PRwNa#S-Wi$t_i+%EVBTfZ{;%;gWkIWQUtt&@XYuKzY8eMqv#NMhFGXkvf zXTkc=Ct~8NtK5XMC}+`^^NpAv`ZX?RB!m_u1p#}YZ`pXp*SLmupZHdPb|6I=9;XKG zSkiC7l#jtI!0p2aAK%ouhDTZtG;ta5L_f|sxt#gye4|CZm(D5N94;vt&P~e2GKmJP zFj!zZCtie|G2-Uh*L=W9y(4+V<>ZgXgTk7Lg~JaXDj1tpAg2rI9}Gt5nZJe`@g%&x z=i56mag0-s&ot&uQP0C}VL;5T>Esz)++up^fzDW#3ZTSWtCR`WhL(_7RThlA^7Ws` zdA~~qDB-W_L9iyo?WF`>R3e;UrZ%VDoxAH@p}3A;o)RQbbv^ohYTsQt{s2b7%x7@m&t9-g``u$33)|>!wrn*UuIM(&x>VON8 zv2^%__ibY)N)z0#0OC~ZjE>t|rgp;yjD*b+dB4MMcTuTAf4g}sMGR1e&M%D(O0N9e zQ!^YMnWRGO-Ce$^%oCv0ZLSju4y>eDU9Z;NZ!Foz$#urfZH+@;Cu`{P&aWH>Lg3GW zTKl$xu3G)NNIlwBO^l|%!~TPhC%s4CWD;IB)-GN$n2yNGbEC8<1t=-Qj9C>WZSQ25 zZJa7-97W{)iW2Y)6Q%7-mmXadQS}woq5ZKpGr6V9Ls|(M zc=QxE_sSmQc60&#;QRUuJ_4c)Js=$;klqYJ8g?@|#>3I88Nl|?i;~o!k_SB$_L{?? z=W4aBSje4Lg~X{f+YgcUQ(FtO>)z!am~$JyB^}H+ppFV=AS~iVUAzeJ;cAimTy2NZ zBN(+!R(?to;sB9WQJs-+xxpg6+#`;r1g|-~4Y_AW`U%?55z>(leH5-q=u&?B1rYZX z9V*cUu(AvN=mJ_?u98w7ZqzE4W{ZEH6iy~bn;po6zVaeS_fdQlyo1X8G{8mjFI=zw zdbs0TTQ!36eo&$L!M826^L8pu(`=Me8wCKY#jt6)>m7g&^mr~XJz{Ug71tl7?E23Y>3OOEf%LgJHwm$3L*tf;@1HUwia<0=Ub$4{F zmJcYEgm+lu-bN=|KX0@rzRd9ei~)gsVS$1yWN)`HVeq!Q%!A}$XFHSydlNKkLbkmi zm+=+sNsrbdFW>!{s>#7xdFOt`ekI=TwEIK|a{SB!*U{>zH6ArK!|Da`mVyBdjBji! zp|DHrbqW5QG*;-?tG1r9LWYbG+P5Gl-+uqwu4_gx6i7wyeWGUiqlQ^!2x~hGi@x?j z=E=`@1Bcyi?NnU!AG~Hk>-=Y+j39{})3g7Cf<(v8y^({4rW_Bx z-}&){m@uR5dQfXI&WbBdG>6Xgh&Q)`0Pky{yp+O_lp~+3AHl2xb^u80+srpTDA}%P z-)dMiyo(QO4a#&k=-IKUNm#KGP)H2X#aa8QlY5caa*(NCbav~J+8)>E>e>+`1uk)g zDbZ&Uni9FX>~^e18y0_go=EfR+qg)KnU0yauTFMKiY}|UTN~N9G~D04!3tVj?*KHk z9i=Q^du~1;OPwoVdi;Ej=c(1>$7n3Wg*cWprrG15om$Z|)w@)~=71(MeyjI8M= zL)w>dr>`30I3Rgsf3B^`72SfR^&!-zj@%GzQnq0sokTmu`hFgF$NNpfR>l?$o)lp# z=I6d+>(?pyd6a3l0j1Yn?$wFHMG&@MnQFBwV8vKkZVR=9Ep~|m7xIx-XAoQKW zi0Kd?yZqe^55sFu^nJ(7ghKZQ-sTcI^d8+hfAK5fc>|`bFgjfPCAICf>=!Ne+8@6~ zn#ryyvj^G8jsF5zf`3NTs$ZG~e;%0_a4}VV7HiCW+)k!~kHgO=e_T?XB%U1N5XLZV zTU(T{nzmlCHM95zCkOtWFX|12 z`ys{EYelb;uPk5_rrtXeYBjn{t8mx4E`_yei_f2if~ehjeaNoiQ7`&Tzj_~oaBqeP z&oSUl*tAE!H}Zf z%s8z*%i4@IYD$9hGoZt9Tu>5Q#THM=v)^= zT}JAG2Yhndn3f!d?C?<}l(nf)MV%IDDy zJM=1EkL!@-i=nx~w)c5<(%G|l3~32dcU1;E7D1~9$vHi(sQySBxodKr@>yPCA23r) z3*u`RdoPP;0Q&)#lwMD+KYNcxOq=I>?TmY-tw&Ai8J=iQ^n9&W24Ove(css?d=0k7QVu=qtJgkXb)yfqFZDo5ZnDZ;Bb!LtEQ-P zaC}s-P~n!77am>U;AI|CM2D~}kaLHtt7o<+Y7!Rf@g*$8ckZ94y@%bOmqjwH zNSCub;8DyLto_vVtvrayHbv{>^(69_vqO57vfM5sidgOnlL0Fu++M9OiNV+7edsUY zklt@}zAb6lRpfBL!b0Yy_)irBmOHI@QDZY9M<4EwThB34i1@y3aFL;Sd?NC+Qh0;H6OWpY# zIS_2fwl%Cyo-6qkfgTpBX;$-o4_v&c>Qw<|Zq=RBGnqrnr`Rgza6ZWHLv_aCkj(8% z&9?Lr6L?!7bOm#8G1&Eif5GQ3w0WzRf9l()ShpA=7wQJ}B~>H1hX2XM{|W3*0C_Gd ztNMTHoR|j*h(IvsQ$GJ2yt#hHb?f?SY-AG z;Ku3TL)ufmR<84PwfbV=9mt;b{p@b5z*fwkKl3qW+V`i08{R~P`0Qel7UcRhjQORo zvWK&h(=(A~s7-oaMobQ;fm$xqcmm-NKo3m09&xD1FAKXg%+g)Qw3$@Bn48zVts<}!4X{noGNGS`c)!n$JH1?F`N?NfVS7y0IJSeHsi<+uxqDJ z+gnq(BHxv~9>_7@bSF5kr(oXfE;n|HY*fBl?s$x_sksZULnrvVj7C*;mTu%B z5?wR9a-!HQm#KDEiFd%tq-RY-r?=^RH(9Y_)#?*4(LXQ2yRc~9mR_Y!5Lb5gdj_EvkUyX;LVk~GsN#Ro;Aw9D29A6o0_{OsMd0#SW#8K%;nr2bwUt}X?7tF zt}u)O$(X0%(>?KK+gZEdy170S`LL(E(wM6D#{@|T-F>=OyhZ@o6?jSO&N1{OQ!RYZ z4fw+cW>&^0V?He%J@CS+@a?*C#8Y(mN9yF6;^lYtTdVGVGu}1BQ+vg?3Lt~UzJ$^$baJ=l<< z=C_f}IEdYRA5Cm&9_pZ=paj75I#ZzoC!F;VuEHPxfY-Wc}Dm|yw*W<;M+L2Q2eSU;^9QZ(A~jI5#eCOd1Uf4~%J%s`I*ifu+07j^0a4i*D53-A zurcC+Zf)cefHngrc|-M6;PI+JAl!4ON zi4L*j4UC?$zhWheDCK#|gi@8)I&fNwndmA0f~Jl*=imG|ZW{nzosZ0Cu^#sSrD-12 z{Q!55^9~>Wbv1jYQE#>guMxEZ`2(>=-t8ekfZhW%vV!XqV|F4mvk%k+7MO z*rw(oS>0J*B>Sr7?S~Z!M!l?Y168&@?|X|kgql)F8<^5r|_OT#|-5H*0H|nACK%;_A+q&*tyCfJEC2LjuOv87lo& zQ=eK?hBq@65KclpfkkAQ8E<_ZC^ApG?!*lqeYV>=g9P_28Zx4S=2~7ab8{sB_mF{1 z@7d|gw4}6&$%vcGmz!J?+m)r(Og_SmI9}?&pD>10Z-)!=gnhcr920*kv)}sXtyCI! zwza-vm=6{o2_*#b_P~mQAIdKPnFb5|`mxZX*${6roke?Qm=wIQ%nzx!Z7-(BioJ61 zz^uREHoJl~uf^D1$)!QI z%94UH+uL0qiu_1@_vXV_XwHiHWuYo3zk5+JEqVHUfbq*5VMO25U~z^c>&e`8_X@K< zuUz2tCL4BHLCsyy2rDEEyKl=Fv4|GT72a>?4BP>Gmd_pX8$YEQE2ElA7y9|gKXIIc zHWd&?rzpgzHrP4qo3|!o=P?U=hQ{i%@yS7+@UrB3NJZ zluk^h8#WzobR=K7IjvI+LD%q3l^JtWK@&Viq%=K&h(B{syLDp5=uZmuT~sm5_=v=e zh+>GMR0{v~#IxTnjj0`}f8wF412^XAR`u%7jK}0IsGq>lxZoP$i6@nM+qZ#1ofLC1 zPZWcO&#C)+V-ix*#ct3v#>O~(F4H++=)0pfcJ~ViwM?!W0YZ17+Hh=%x zpMTS9BVdT=#8qgn4H~g8rbmEi8^&YYAcbdCUWzvG$%9^Uppde%Ujv; zP95U_IMFq&DZrU7Q|pM(Gaj8v9Aeb4Q>j?}W3R>rd)rO>e%|1;$P5aV=Ph+*vRvCV#OwMXisXf^ug7Vo}M%)+gAFAd}n{lGAq^gu6B zD3Ot17@6u?8_dL7^T$*U!Sh08c69g#`KJ96pJ2*&j)!Q1y#a!|b4w?+>lEBblEv%S z%V5-$TQM%TK$Nw#XbIWrmCw}h8OAnMC5BkCpd}wJYTuZNbQFyh1f01Z6C~!YM?TFX z6)n~CuY&l6Z0H(uds7rIh$)2A6*Tw`00cVTxl$Gs{0xG}6RfAvt?ddY?^gFoACsXt z$eka1J!R1{Em(+y3@yNL1IgO&laI8y`oFz<#S|;R*}Z@41)cRc^F?~+@l{Y5tSp=9 zW{opd%vzOSztw=baK9eUaMRqw)*zP$Nf&CDI+=?Mh1WiX+xD~-A%#;l%k zcq^m(Xv(VvRN#{`|J=aL{vcjSN}Bqd0*a6ByI~`L&bL`_E%V98IIS;hSa_lk@A(Ua zuC21IO9WRojpJzBC8mOpV`H#aJPjCE)z0<9?M4qgMqz(+Y(3RKH5Y>xOMpU{18o1& z-;u=j^ourNKu-7r@H+v&G57hW3o1lRJ6b|T{(D;5E`cpCeE~w1Eq3v7@IU(hkwE+( zFpv6Up5LA;S;bIL^<8L0g!*p;x%zxc2>N$W3b1NgmilLfdjXQ>3-AR9HP7r~!7W!- zZ>$n*o|hV*>@VKOyfeo)n>OOr`=ybkppE_nF5iMs~su@LBH*$ZlT>lPXSX37+}HW zD2pT-QuE{Bg2dy{X5=0#y6P)(ArK|0W?3_jAVjRC`pSsXcZmW5cjtDZGAX6GitLzj2;cU1!|uL3#)-c zL7W5OtQu67B$%9)9k&7S^AFv8OYXRX!%n8jd50|n#a)@x4NjB3or1wb+>v>sew}L% za7c^A$gdw}t(GhUa#F$pvF$!;!zXL&LKCXm7Ujc{j(2?6YT0Rq8{%j*_aaGjdVRv= z$eC%qr6+7UgDpZy3%A~qnOtPY#^>b5nT5Q;?+sDV!4+900c@iq~`( z6jfSZPZVP5^YWhvc=5o1JcyhS+T5Z>l?$`5-b6Mn`e)l|2$v$89vMo+RJJ2Yc0;t|o-`4qVU9qt7bR({)y~-5MmvoqN9|3!aQA1kkH{Cn_e3$?;#P@T8`>wEvTtSW)m(Z` zxIZCa)^bLTPrTT=g)gcw1$IfGAZEXRtDi!H=yn~<Iy*D09k=k!u{AvFF;BKB?*Df#d5X znDtV(w-^<%JxZ}(wBoK@)x&Z-_iGi*Uy1dS8!vX1v+k7q?p=FNv)_vAOcF307_xU$ zFZhwnzG9I+Nk$m>Fiuamr{i@%1-V=|8iHpS1-52@LhCtrVwJzfwdX(6utNJ6(^rs$ zWyUAB;r?GY5+lozyIxW8a$PvrjT6A8##wIsl78KENh)_Hlhw!|o~viZt@fTyzt!XN zAI~+H!qeeY;+&S#jhNe66NA^t0f8DrD2)@C8A~`P_7s$p`rp9H!_ckDQ%)bztt73i zPhg(#JeHne7L@{Bq)yEc%0AV*LKd%}7DdDVUHc`_6p=b8g|Izdr_P4mPkI)*zIkmp zXkW&+G+&iQ*pAeYpVnxII@YYf;&#BZYD>!+9cdjDx6OVMdlMo9f9|9bMpouP48zyg zX>xVbm-|DQOw(W{HIu6Wl~@Nto5{&Iwl;V69M7kq@xH6}!s%OhcpzYu|zEsM^ zo%Ov^YN2G=r(oo*uK{=udg|7O%?{YP0ZoY!XdP7-uz{z_ls|!Ky=BWl$b1cErY&}d z1k+vb|9Vg`i-Ls1>Xuo!WtQq#y2al}!M`W3Uxw6`yEJFutOw;y{b!(AUK@(|z$X(- zN)W9HTArxl7%9^_SQdU;c7B%=gTMVH*9!gozqKg&hZTr&#J`C@?jHLu!T8xLud6Kp zfX?TP@6sA}P`~OPZ9CC~Z=)22hejSib&0P{b;N-aBIp9>V*?i1ZZGdo2M`{zd_~Dr zzkN-4|EIZ+n`klBoKoG{yjwh%Bb`?>P@7pZ=o5vRuFsXSj7idF;zkqJoz&!;znynhLFPsk@PUn&pEPwl9%#aHUqMFvX zxw1dI+!nvIukz3j*0M9D!xJ|>rVkBKnz);0yAbwb{y<(s`Q>Fh?|f0Z_Hie5?2UO*iPgy~_%CcvCj=XtIv*Zp&QOitEL z67IwGGX)XR6|UZ1puJTfn#F_kszK zQ|#`Y{+s(LK=Us{r^NJTxS<9xC-wr>8tsbv7bvU!-As1Q6;jNXVL`=MbkeKv(H}cc zq`U{QvgO`q2~GMVPb#jTl50O5($~$eJ)bY-t*>tu)$Wn%JJ74kws`ftbUC(|VFGF$ zR6JMyYySi7Utf}#!pGf>&zBeloPqT6ssX~IqavqDC+z5kd~QYex=c78La1*1xty!O zPuJ9J*TPJ<*^ReKU!|TGPQ-`#L_j03-U)0&H9{CeBHsg7%TDGzQZ>@*TFBY@PWbJU z=s0FY#+C%~D91-JM>Cc{<|xsD`~AFvERl7Xa)mfZHVGz_YJln(dg*k-r#TxOyKVre!%*yY}I$7q4g zeT%;@iu%)`-u%xX_=$ja2ahGBPJc{fUS9743w|+f+tGmU^_scNJ&EC2j+d*BtAYZN zk=8s39(A}h2QPJ_qyp^rxKP?O!kCR0%u`wPQT)B5hFA)oJ` zpl_d4$9+%oBL#VG6#Se^JTG&r(mUKjrQu3*L>9e6!`9gq6!RHd8D4`Z-oOhp&i04m zH(8geG{9Dn+j-MD(W_qT*2XAFpLU5o`%maM(QlK0=jP7yjOZeJeLVia0wEDOf$Aw^ zX~oTIOjIVJ3tZ)bZTs3IXYD=$nC1LYAa^hWnOJYSI2mO9oVs%r)L-*?PT4Nl3*$fa#{??0Cr zVXwqx!Et1A1KBzIT8#pHr%!M>qY#6AP{GV|6VB6bP{Z*2OeRcn=YLrd6nfz^A!(&x z%^gOuAWnts6ye;)P&aZ{~zA?=_-l8lPPXb+5tle<7xc(VsV4Bg9g$ zZyVgFZM(`~L3LNGdeQ#_WCOe_p18tI8mX^%c^)njXZado+(*6NvhJ@1 zE8kNyWTg1^`lh(Iq?RFVYtA0Z8MZe1bt+Ka+5c@%2=5rSVP&0T3i)k=CV=H_f!QO0 zUklCSE~jJus9GtL;hsp=oRJ1aOI=_4B$}*_`lvRpOfD|h4L)cg{rA|nQGf7o#vj;s zQp3b>+h4J70;f&Cm}|;Q?ouEQBiAG?4_hv9!fQ<2V-uI3Tm%EdKmJDlftGCTyk_m= z>R}0fLw9Ignx+J&7X^ z-jcuL=XK>g7s9+Jnfy+WpcDe7q7jwqQqcRZQ}v&N5*4ueua*uiY9>^{ZByx*ZO-e?Ay$&N-kD>F| zuR@zplOO*g_)UOzw(k6#Wmnf8bLUCV8EWYnd%db23#TJuJ}~7neXpObC#Zg<6gyvE zM-kt8pswu_0MKB|PG!y_wFWBt&ikxUyUp+O%POoTUMSz`RO!L02x;~iKTLw;oxDyD zNslUyHw$YYbo7lRljS&Bc1uhlgu%|Y-;#nTc8$Vvar9#^wl-S*%(jSnd2~XR@|4QJ zMx(fFSsy&U`=IP{h25mB*|ye&Yy*S zQ6W~Q@FO(ln!_Hz^-tcfT90}x$Q=lRO`qZAxJnB{X`I-!>R>y~|6IooS%1U7!Pqf0 z&1&CI2ig5i$u0v=q4o z25hq<$!$QCZNv&_2;4migxF+Htt78_X^8VdWnKZv3&38Wcj8jh3l4z{#I5xCkDU!$ zm9x!m&Qa<6ask@CT38Uk@S+pBIQv<;j$>5u*o||MUhWL|!E)-Cd$=FCdxw=-i}1=Q zO1bvCFRuG(dJLnAs;8<+-IlRtg_Nq4Vk~-*5q2zQwR&4UHH^7<+-f>`Mg^x^v^my( zk21*pfiKe^diBSh=6p10a-iy*=8ZgAk7^dx7DW@kK{*X}d%dpzjz$yW^Lm^@hH;drbJPyk%NV92kE z5~aWXCXw}^ltuZQOg%~O51G18MZAxA(%-Ap?E|ua6Sy&|F<;8TOp|-4#3@-bx#ova z{?UWp!TnZUlF?j`9k7evD!SDi{X(Rc#VZmVPolaKNACf5Se``2pByfs;G*VZ!}3FVRz-Vsvp-G#!gK1Uzm>V8*0>G| z97tnuLjAY*y#qB-^ilqL9&MFqbqPi!*XEDu9tKC_JKQ*H1EBfE=FU~k?{S+21pvVd zSRd%H?O3+({X8@Zet@*&AJTJU5uKx+VwWFj9cRoyJ>P)J+z}9WvUT7(e^j}9+8uOH zpQjxnk$e(~+3SrOKonD{!a=+d4v#y`Xr-Nmj55>(J z(b7b3tIQ@~UD@ZS2399zVVt4YOypxlkfQr72%MN1TudL$&VQ5KbrlNv*p*aqNz) zPs*tc>e4OT;AXRzNURuJ_gY&M@z}9Nb>Pt8!e74lU9R7Eya7?r{T3U~A_g^J2n zy`9a%dEOkNEF@@YX#UFmc}tgC2qu4oC^|HIqHp$utn$XEeXn!3E_}O-%pce2!zwQ-xneot2@L2wO_So^g|axA-vD!Y_Z1J+hj$58Dg|@%8I+%i zJOCL9Hp|HJ-Q|z!4zyTMK7X}G#SEN@l(}5`r*!$ghg085i5*JZ)Gbw~lFsZ_Er4zs z#Deu-|D*Z-kLLS7n(y6uZU51H|3~w^Ti*IVn(zN;z9|tVSR?=ctNEsE5GZOr5$Zot z;C9W>Rrhe-ZoRilU=y*2B1j-7R4ck7yrX#aZ@|gQa9QK}w+(R3h^T zjyPcINh&y8r&<(E5s*VQf_6`l{BXVoN8bmPu=_kow#FIv(NsX27!e9N)w-vnkT(P(&4goPw`4=!-FTi;s>O+6MK$fk{;Taw$7Sq zP7jf6FBoxSfW_W;U2U1EptKH0pX_|2vd@pUn$PkD5MDX#L71Pqz3<6cw6;mzdHm>YAOV)l)@Co-74BFM7UVUeEI*PR*! z0^I6t#~#IBp2l0-(b(7xBzv^qeqjgkMB$^V)X4NmEH8;LEj_DuX|79+G%<1NsxQG_ zD0-VbImLRQ*$DrWeuhx^VW-!@!wIC?PJTo zbdbxkg$!>{+aU%~fTN9$00aPU0vfW{^PhpL36`d+#fFk(>YWvos{~N97-yF4?Evgx zEL~U{%)(3lZh2Ph0Bk~DkV9o?8na^98rQ)5asMp^s`0%U7nVy2zqa0^TKcAkYP%y! zt=>FFeVZUk$5LM@4DQonK6a#+fcj-=5}^c&{t=>Sg7+C#*BJQrz|Izv})a7odyY}4?c z%DqO7YdN@Y_*%ZH#SubSV`!%T$N4M7(q2(LBs`=N`7oK>YRNMY>ALAn0#L7Wzq_$u zTWijTqpLmN8!_Lmo=s}p*3X)+Qi!mZ2QC4e=yfTuwFX%m1mt?=MwfDJb-cDRZf)o$ z%E*bgz>&bq^d#YNNTs8#{~fvB>Ps04H?jH1ZNzZ)td?4R$GNg&Q>yK+=9M@wcS##9 zmLrj#0>ka`u8h@OMf$9mb-WU4>{4g2MbhiW_4O+=&aZ$3&JM)8pspOa=q?rFf2s6P z&EV9HCA-P6JlMJa(}?yzs=v)PJ0~=wVtf?>GGl|>mSMIO<1osT>=Q=Nk>csd_YAx> zxyz65hjmi>5N@K5v6UUT(Yn!Bl);?V90#h6@ypY$Su~w5C;|qfzm5kw2a}MvJZGy7 zQ4~GKHmP~D9C?L)G@(5yA8V_zc7;o+Yk?L}OPeFgrNsj#jwE+=UY_w$YkxI|FVBk{ z_R<3`+I?z$lyoz&qWHh4{;r0IKk@|;g=dhE_lCJQz>MmT{jUAiR+r;S*k%w{6Q1bM zg#HYp{23}BM-F9rvpzycWYNs+RpAEAyHebG7c+e&h)6KJi^ppGpx3<8R5hWij-FGDoj} zc>@tKrGa64LEb^#H%#1%L`etGk4zPK!gNM_Av zn#Xo+{4Cnq4ucA96h?poW0bRqfP(d5#@hW;eaF+7FR}#LU;7mkf>agwDy>gDRR{o+pNB`pB%d$0OEfbJ2oxM(%ya ztoiGfvQc!adc*t`Pw{ZdLzD+TrD4WhNyrFh3T)2iorMg;Hj6G&`GE z6O)<3O0DkQPGN;EB*Z}2B#E>TnLi#+^S{W@W5<1F1=%|~aN?lHZ+!vPPh_nkf4R7E zIUrM-kfx*_J=?fa7`|C3HX4aRBNsv8H0XBgd!gBq#-o5b@^SEI+&kR^_F{WXs$?rS z1nNo)thf!0`gxIrExIEC&XnHqe+Q*k?6?lA?JYQ;a=QVuC{+GasARdVL=;DSKkYg>6xl0JC)y=9NQhh{<8qIR!#g?Si|g5NNEka z6hI{WcV(+@|8`K}|67{ie*%D-Di${Nd1HSq z`Mpcc7ZcdVwe z&gqK)`V+U0O?2=Bhc-JxDZ%p5Yr30>l_{t<$W0L#)vwpCaiCX4693*BzyyF{2Dh`yXaz0 zKH?DD?%^s#hP{wP3_&JC{)*sH9cw8&cw+UM7#0m<$(%baxmlG|^QBD){k!S>7qRy? z=0cs10m6HzbwXtywV>mfPVt{lh<$-})$?w9!ltj6mb{_!d}axm&W}@F_+igRyS|Wx z7ye=4oo~#p;5WNausYHBoLH;6W|S>ydb-$Qs7NA88UMA{+QryKX0LM%!X7^6r zn@*uM3P35Yn21F6iWyEdtcbnF%lOPVA>MBHlQokNT+v@|a`+i5B{aGkOgMk4TxTE> zR&!B6PL67_Xr8{U(xj^*fi*BnaP=mX2UI zHL(04+(1S{b!Q73xP^J(@{PlAI6J=i*-Qv2&u6ZeGtvq(oW_lue#KN1u=RP%$*`@U z2^oIhG%-R)U*k?h$1TG5V(tZ0)U!zB#Ja(gsdw$g>3ti}Jj@DUwlG7VJ+E!5(ey&W zjpnMMLdMi^RFwmV{{1oCV+c5nha-ejSSSf}bbsheHnmf<1 zCbqWUW2GqI78I1Wbqgwp0i}r)LEWHKMIcBMBT7jCg@BXM(BIoAVQxtzped54I&x!7vp2QRz_P_JJ^)xlnEMu%*02zE! z(xATcVFK%Sl-~oHvUz_OXNWhhC&*<~MWej!{t<-457wbJz!lBkc_`yzMAdEJ$xbZ) zQEnx5>^nAQ(kLG`bYb|;fJLBFF4`FJ2v)2nOZVBFqNn*kLk)D9^*3DcI9u>>@pF7W z$+X8tv^1n*Huceh1QMct(YB%0a>^QJ3}T7r*5if=B zQ_7-a(bjJ*k5dYd=$ZG09+}<4w`GSIesL452EU;K)Si`Hcb@Wko?1A6eec}szK-D0p-5GqfU?LpX-yLq z{UacdmQNelZUCk2u7@Jtlr1D|ef)s&UA^u?kIyJ0Pq0Hn&YR|$_u1KlM*16qytFx(+z+H@7CNmBlSoCBZF6T5Ab|} zGRrW;l3eS$MIDgEEDn%uwrd$@cn-Wt_r4;>`Cv5IG>WR>bgTx7)wfK2i5WDS=S0-& zf_MA%*@b!i(2J8r$7Hb1Sk>z@M*rxi<&LY?DrN+-(c zEc0@2KTpxK42jKEet_F|Ml8>9qfzPe2z(9FV%oy6#+*us1KtW`v&TrUAdAx#r47iN z-}-}h;(@DxvimB~cL%8|hIM@aAQJi>K*j5afVD|`64J*eGGMRNH+dvjHgE~htXERj zISHG&c;uJ7pl)Y5o3$0^l8PYibZ71>?ORMByWLqwDt=*vYpG?hKssI|}^ zK$N-eSQGGNAN*S~k0{?l2Z-`LJtHVv?9vK&G1-}(DRw;dNa9OtE9lQUMId!zcN37bsfLi z*g5rsuS{qKE>{X8>iNrA+E4~!m1dZ4bBa%e2F!Zv`zJS~n}u$9i1u546?0NCTqD$o zvJ{?e)1A%vD`;|MQyI(qiKOU(frO&Yvxq|uAx3tlA!=$uU1cY8vZzN_&C7pyH7=Vg zOBt@>Q~4Bk0i*Um6nR&M>Hcf^d1xG4^+1!Zzp$?xCms8ArZ5_#v#QK0z)_1vaX+@O zH0PT-w|in5d-U{zHdg8Bq!sGsOU|6=rgh7D=%(|pzRxSGAG%R$B9OGhAztv-_3(Vf zb7J8Q3VOA>*qQA7VsA9u?9Ef*T=?d0TMIAE-PLJ?Pg~lsd9?BBldxpQg^q|DwQ2@r z_ShYL%%~3lU3deyRFvy3B3TbaQPA1e<*Q<~j)8;IgYEMz7P@PMgaJtCit`sX_zJ?l zgRFW=2svr=qILHUE3pVojol@L(ptzp+Lx-qC>w=qC36f9(Q3ltaCXyYTAs(+2)X_K z%+*Yv)K$Cs8pKu65JnASiPT~)qrIfZ%CEx|@#}N#kh4fqdi&-1#$Y=hTLY{!qJkmA zFQ0Y*U(op=O>p4B~H3QzR`bR_wJ#7oEs zzKY#phkruf$3W=20|0`q>^0fr?%lPN23A;WU=m@FmF*x z?YcwC5x0JH$P^tg#?UYI&R(y`nZMmnv*ds-3`8CU1k^S0>3&~8)Ek}Tg_bGuKxE7i z4DI+#wFV&Wyno^YcDmI*$$I{DU*UTRvP3cwbk%ze*<0v+LPYCGy#S???J)Z zPWI=IcLsA4^n_A;4YZB^5uSkrzc=5XL>l&9{5K4&Tt+#}a-rO<)6y)Qs^uy%GVDD%&eT2z#1&*1w4X zRo~igns(Ho2uv$OF%*9ZTLd#+`R{}d)_W9sgFP3wGt(`jIA-XAWv

iHkj_wIhZJqa)^D)?B}&Yt6R?Bob$<8FGB1-b5DF5Meo?>XqzR7atq#Yg z0=iG*FX!dNo;-i>NF+jgH13umrt$39((eVK?gE8=zeAhmN3Itfg4du1e9t;dw;qEh z#s}1z9;9`}rShsRK&htngTAUsHOv0#<%wtlESr$~!upu&w2{j-KTRk#OSB?>{~CmH zI-#!7Kf`CU5%WD4vv>-ZVBO?Qi9$W^8!ewoQr~&o-|L`U6>{lYyMCXMsT8LeDX;@*cMiQGAl5bDZT`FMH&50xb3O|O z+opwnt~?XrF61#I`3j%LwMIlwuS?oeW$dHAH;E>fxjA17zCLujaPcjoTfg=cbiof8 zE0bq~g6yom17^^HraAksV4YT8lKvLAz7i_F!=tDgU z;$csT`r9QyMeBksc3NwQB{Y7!-vIk9B6g8F<@>A)C2^0upWoG5@>N?-MJUWf09L&> z!d2nXarOb7t8Nc7-*W-p@1o=qb_R@K|z>e#Kiva zss*w@`|($6Pex`*$Sg%bA}*S1YMD4Pn1tzw5w=SU&((p)Fm^q#aR7wZ>L@`%ww6G@ zyl5hW@Txk3i;zkfNUH5rxcr}h^O$gP87g_f2(pr63Q!r6ZMR%I_~qIm>t+NsOYqUF zFe8D0XTO&K#-FpAyJ|Sm#!>EtWZRxQ#riIlFA{qDC@V`z|1OaS?Bj?VzsPsbgt(KFwtMzf7x@WO9{Q*Nc*!f41L|-Sibxxlv-si7 zHRgTJe{m@X$1f0ak-XhpJGM9HsfgRB9tFWb)BB0rRldkrSxZXrt6KphZ9%z5b7CZd zK%Ms?-+up;RnrP!W<@T*1#Y#8$qA%}7rsJR$d}JJg?RBGi56<5wQsq<*2=a! zY1Ad%(`qHZOe{XfRfqB$=m7ak+bvA_s-mnb@-ajFq8z0t69GEJ0|zh}L4-5>UIO&! z3XYL6x4OWm>`8=0nxL0oWnwwyab>61X*aS$Z1MAmn48A?<{%VY((;uuv1(TTMwD;w zLF`{SZtvhUa_EpEjE9o&`B5*(itmv*Z#J2g(#qT zbb(H>ht2@Z4a=P`=2#*-EE?h3aHY#yg{zXxz?f-o@1HqeAYmC4IQE=hBhY8EP2PlG z-Jt#}Aywq96J#Gl@Eai~GA_ zx2c3_eWNHEhSi%O8*1WC#rHwr=^KB~JzFICT6 zC-;zj$^^^SO6#s|LlrFMBCu>!qSewrHJe}q(T`)^dQ#D2JuVB{u>rqD>Hwdd*4m6 zz+L;suKel1ynoTE&k!^QQXW^%f*vu2Q!BoA>}~D2dIQcOPd4JP7`@fbM}Vdj@uQEY z^3(w9usPc;v0hAA0sE=CdrRvF{SrB0)-(W>&uusUo6Cyi(n{hM7VA0OqbSG81@| z+eiq#N-W+*0krVEGKvDgrs&VE9cQ)g|2DvHXd@nta{WU5gY#r;6@u zjD)tlKY~3FEn&9hDeca{opSezVS>nA3R{nr-eFiR_x%TpIDP-!mtT%QF-K=gd!rE_ zgN?PLxmD&g-*9PC7*L>~eS-;TU4Z@)HpIir?mrhf^DgkN>U={C2Rmj8<`iW2sPl{7 zrS?rF)jaMl7X<@l0ZpS~TW+~AUwBW_WmDG-AxkYGW_UZ4^e!5Oo zDsW~(0tes`3%6^ZiDokI^i3!x%KkA3Ks|K+$dO)?TI+KpX{6AWvIf`mz z;YZkZi|pV-ZSx=)2r z@BwM;hSqC$bQ8+Cin_PR#2J>KxA@vsT7L_yWleol2{%7ozOh7V9g)W7V;6P!_tG~wGCGOv`|?T# zEroe(&LqXi7f%iEm3(a$6*TA6aWnSJ{bEiAr^ShO1!sXu@C*>X$7Yu*vPALF)PzcGMNb@O&AjaLSHq0x|R|C*nxh^hI^YT?(ubzt^t!-6~?bZ#T4VmKd?gVY&8cS zxmpDkg%ekY{LY|(1-8zNAm0lD<3L&6C!U@ymS!WKs*5p6n;RfiG)+h8t{mNmxOj}Z)t?n+7?$ryGxx)Q}mOZE`M01@La&aShKc>E<=Z+y587G z1-*Sz*axsL4knRsYkR4Cnqncx5v@iQg&$g|nMLylpd}8D5qp!hT%Y$#=17DW_G4@3s<2`Kn|U z5>J|!gnre`Jjm41K4g5lTU(%bYk}duN7tBw2AAhrZToL1e-)hWw{PP>bypOM?1Y`Z zq$$JLQ;sqeLErbkm(2wC4rtmB#RO*EmEiD87KW-CT8CBP#+q`X*d zFo4-iTmd$V#yrKai(^>XANT+HQmE6GfHEQS$HT4n_j7IbmG_ zXPH>pR1tpaYa>yDu2pV5s1Q4p9)N@&{rr(NJL8f|qO+h)0o)e5;Sn5rQn&T4chtc( z^|xz(Jf?Rv#OY(4gD0B(*?-@ZtnsjP(I8ah5AE?m>FcHjmYT623|yl)0%cfV#xK5V)Jsrv$Hn|57kqFjG;dD|U5)@b72i9q_*EM5P%?zm0;%{HW)*YgD zz@F?0k1Veon`q*OJTm-8rtaFDFS4Y=9p-yTH!_R@b%?b%cCha^w=P;x=ySq` zDawH)PfKKJ4+dj3G2_fpV4~o|pK_5BJ)u(@@Kq(}eot$Z>_?yGz$QQ z{z^rXtgS_PCGy|5?_G&RbX4Wj(5UG{Ce7$z6wIPr2Gp|uyVH+4CN~wFvNBFxC8&VsFI_ahP+@rE;r{|#hEBf# literal 0 HcmV?d00001 diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png deleted file mode 100644 index d5ebf3e1e7c7ba540bbd3e3368d6e9119baaf6af..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 15091 zcmeIZcT|&I*EbkN1VNhgX2b$0(nJ&tMM0!@0YxB2P(-9ds7X|s6e(gs5F%X^NdW03 zbi@KuB%y>Zp@f7UNJuil`+n}{d1uy|HS7DnH8X4G{UaymI=Rj{*R{{yzq9u~zr08p09 zy$fMyT_5r_x)}ff@V4!J_I3HZcLM+}JvF|1<$8$AGX7iAEDlCk4S62QhT7flGJBt3 zzQ6J(_j#honTz9xRU~s#63R>(-;aqMyyt#*>#nU)i)4$McXeG>W%0FEwMT=ZJjaU1 zddh=dleeC<1u&74-IIsPkH4AV^KEuZ(b96en$1Ls+2QM%yS+@}{qqvfuI2SrjX;^J zKxPi&cK7pBrmxQ4EFYWpr;wQpGL>WR%8k|hO;ITbm(%yB$LNDm`Y48e%P{ZHb=9vb z8Wur!ptJqy%ewow$m@E<&TB-UE_$!;UbDAfEscO+xYHv&ODcLUE&VBOIZn_Abm?jE z$A3-h*6p8QDh>UQ$J$P2;>^8t!^o2O?(R5ma)iye+yi7o9);c+u6xFZoVgTWgXq}G zBSltY`d}+elhMCAYj0H1mVFQ(5n^EYa4eT#U?6P^oD7s}weZ3yJL7h08ekzic_f&2 zp-&*0dH*ZpU#}TkHAbJii0RW!RKoZ?`PMxUTo)WT8CpbJdWgYPjS{z#S-sB@DZj(< z7sqsbFcYw~QXbHRWzXwuEO6b0QZUXa;JK7jyIuphn{i056l0m4ft?6-NX!E+$t{Hb zB}~&-nKu^EY6UwfKVy&V1mFnYNR1~WYPOyQn{)2`@X--~$b0*NtGaKW938NvF7NCN zhBkekI_)j{{k;wzS5c_zDFtTIR0X)B{TsnS%w$U=p-PkLNufKqfpq&WSSzkwCLQvtO*M}xws&*qBe>R64fXv!Wp2aFKsHIE9#0#QsO?O+( zIXPu_-(z*Eugoof?5&s?^HjRe8oAzCN)F7u8~}`QN}y6kCrIOTy-)Ri!Wq)_;*p~h zpQluR{#ma2UN6cwFr!@3b0XMMpKE#fTknPr(s=~!iVdQDEb8+tMHV51snD)&?AjKc zZv8>jOsHT}=7&j>-;0G*qWi1q@b@&nso90t6Ei=*_0}m!dY9@*IwD43 z*B1yUM8}#oQ(^QIpo4-`uyv5rzSF5{PVxK%agmCXnveWM185Dcqd9Uh%h~ zkm)wDSz+A!u0=-&)^qdvAP5{=>f-mP{2S6V^;Z+))xfWE29`7!(ZML15%g_;?*J)p zWJ@Qd7~9``qLTq=!1I1b}Ztj zuGWj+vCOuZET*;>;%`&`O?M$N7-Xgk^WCdbuTZ+O7j9>UsqTgNd)?jXa1_}E1ON12 z;37CUxadC{Y+|MVn(FhXD*OZ;KN0j7*4)44RgY?-nlPLHh8nlpj^897=9bSU-`_T9 z`s$X`B3l37;Id$e7-Z=mfdsMN@YM}_~)^^=3 z|NKRlv#=y%7jUq-3;|4OPr%l${eiCh^!in* zZ1u+`r8>`GAk~wqYm8IHTJv$P25p%)0NY$ET4oBwkQzQx(Ef19kSU=A>=ThlQn(a* zgnVMcmVPsI2t!~vE+Q2G6d;6y;|P~&@%xZ_HCk4BHiuUxNE?(5gt&`vB>HZ6@y3skVU=Q$s+hZa&Z@*`}js2IB4Jo>U^`mo-&nl*{5qw9vKZ#tHjid5(xI9x{ zzQ`vIjVLwf&{e|qX(v(Nn@)&gCsuHSwH>fu?IBnBQ&CF+cH`e}&xZ|A`b!9ulA>xs z=%d)D=*Z}r6oGgPiL3p&XKu--(8F@%e{s2m@pd#qSId2gIL)ywn@q8zF&&oPoovyX z!GU~q-1-x*QfJ}qh+AhGfJzY^T_r&1cX%7t6t_<$2_4yn1{)i&%ZC0CwgszHpvsQu z+>d%P$L;U(FX)Hd$nVVoWyqBHIcanqXdP-^J7hLG<6EZQYIZZ$7R~iux*#%sPL`SlzcC@l{4Lj{q?HFn- zjkowM<`DHf<@l{n^1lf`Q87hk^Vh!1v$oZFwA-D zjVQ6^{iv!?WbR=li1Q|JBK9keSo?d^VCuaHw0u2&M%Ec6W;rF5GZB)7X45*$(utJQ zVlw>s9s1su@ormTz3iz?^5#25glh{@RW-v+xp{KgCwR0167Gyx1~zKj&MUX8KfN=a zkmP{PWxIU(s@P5IYV0j3OucB8b3GqX&*r^lD zK_3CA>E2DR5Imh(7?9J=vods7tRey)}G5iK6g1^`@m_CHsaOwuV6MJz}h(epYC z;keQA0(=e7%-{qyrye5$n$P`51(z~Pnx4%EEsS=U(w;-H&bs!jO&*I{HZ^?8dWdhw zh`06iaGIL6^VqjEF(|3-n_<0oHTUvvG(OOG&Z)>D7)s{}@f!!vga)TEB92|Oq=s+L zrfBc4vE&5d_{E{rt?wOKdbrR~fvo}as1w!8!%N0o z?=>a@2?^}kJTX}gukj0{&-D>^?GJ&P+pghSVopN3Vlz(?8=6Qxx%7ttQ}Wz;RoXhH zN!(MXc`sFIH<5a^oVV|u<;bl6I1tUXl^UI;Z^RO3>@##Ej~|4PTOah;fFmgtRpzwq zH`N|1O$)phbz(jEu@k}@pY0Hp$-?hDwDWD2&?BYk;Nq0qO>^;W@yl)A2~VQ9M3Q^z z0~(##pPVGlPg*15^O4WD;Rf%-q|!iL1-$-Yv^-M;<{*+d59cO)bjdrE;LqIXPU21S zMz(77bf`D=@pDcdV+kW=lpji6`@O$=czx6a=SLfT6(h}5PbXrV?jUF9Rcym;2CFwd zmdf-gyI!BKUy#?bA*7;`ji0nmray-XYbKr6jFq} zt-Kdb@|_|rog6U3`Jr}-l=>W8?!X@4{6;&lHv$sr=Q4h}a{J|Q8d_co>?0-Ac^AP_ z#Q8wO@KYCleu2|9IlA1~UJ`GS!JOjL?y!d43}cD9PPelj^YNY5n$`9-vH%JnghTRq z>=C=MG3-qbT=xhOEjTn|zXQ?}X1ul*9TZMZq(moIb>b{mB}Au{7kwy7`r#KZ8RCXh$m*2-!DE6q%x;v8x<^T$;?eRS z6=JC#gZC)dhRN|SqYr+0o5^1_V9*G6bsk>Ry_#!Ug z3G6&Y?W1*b92F=Q`+S3K5`LTrFg(t<8#s0B^NLWOd&0_&;N~xTNWCJXZZo>Ug(w_Q z>Km~}0iTq-9f*O=*QU1I_chy+vh2y5p^ML^$3Jd6I&cvf5mRbCrdx@ip6`k{bwByj zGkqB%AoDVAUs5~Ee#v49E0gK{m)QYJtY*U}e;Mb0RPLZyP3ehfHmO6cynwTjdzH;^ zMtgC>-4QWX699Mqf3!qDvDBr1rcFlj?Ro6b$-+6TYLxVC_C?Vn0N)VXIR}=Y_y05N zUzNyluV6p_{w$%94%7N?kH@ri2k6<_m1j$@8DX!L^F z0QaM$Hx?m=mHGz-TP>V7U$6eU8$81CcHpCLFR9dG#y8+8IhR~C>Gh=H2L!D8kl%$?~aF&6idSi9n|cm_3C;FnfAgu(Hd& zJ5=x8X&2iGy_8}w_A~9A`OfRU2v`uKwRGqyIxXe|U1+|q0GKdNlI3L9?Pi=kkbNI4 zUNg9JG%J_@n|w}VYnEuDG?JrhyDw}y@<2Em!*FqJF?dU%)$OIX{9F-MzEgQeT%3$n zAcYg(v{UgXQvOhzpgtQR_ zfj+I)zZK4QunRNoZoqj((bxG;#}@T#&P*POe$z)g_u~PiNW>;L;Ek`~vRveI4IrT5 z&_BbvXcMnS!{5En^tE1emnXK5HpmmBT$Nl#7O#o%u;-f{fluuxId5Ra*F5%UQLYV!6tJaREog4_RF1CGKxY^J&Bn= zaY)cl#h54wE%zkabxM_j8hgrEvR{6!%A;(tLDhj6{v36>FCJcgLI4Vo+P8VtomiZt zhr6OIM7BwrYGseEsYSh9q}S;5qitbljL+l6I*PQswRgUr#XH2xS6krr`6ly2e@ZIP**N0Xt$g1j z!j8^bI-I%3!l0%?J)^izu~O8y;m$aZIOFaZLHA7h?zayL0`~{)g&k+1qUluaK$S&s zuw~#htV8n<2tX2GxuNoWBC$4yjV0B7zZL+Xby+RIK!8 z+`|)D1weo*;xp@s6mFOAR}oz|1pSxhkZOEKw{h_Fr~&iqnm6*zx^u&4<+DA`1Q`@+ zkid$mwoB^aUJkF<)^b6iB0Y5s`rDXpu4bq2z-3NQ#+mPhMx9k^Z48PIiBhV!t)=%B(51s}yS(+;8aV`F3C-R`Xhye@hE`kIkUO3&Hcy9HKcfW^Hm|f& zz>tIp)yo@Ohd}T#B@J8vEgYinz!_G>hfwrDdgHSy7zf(H!51tpmah6yg9Lb;G1T;j zZyP@+6Jf?(spFVXPSATb$lPx9AuSJFZD?-Q@XPEvEF5Sz8C-<_tx!RICjeD>gg)^i zd%Q+(scNV7qfn%we1Bbmf%+k}n_r$=21UqR`0-Y50psIumeyu*o)=p*EXQ@&MC8mg zeSNbGoksF8pj4dinRUF@Y5`Y%k)Pa3d)|Z{CL9mtE7F*Ed%0^oE~@9e?FKn-a9zaG zPGiXu&^izuQom>@a=k2D#}2J{1_yO2?QZqrs*UhZ)vA=Yp&^4`pSiuj7+YPtF$0SWkI8|jyv$vUTNWQ=)Q$J9cl>zHRbUv^0*>Ch(q(}DJF#@y0ytrU#hu5BDyJgzkU(NWijA5z3p z^3y^;Hq@C#;b0;CV`p>>e7h_UgUU4wXT*eK_v>0J^nZ{ z;f3ETlEZl@kP}}M<^c?kl9mfc$$gPk0JYSKh-%#W?rfS6Jyw``_B?$%5HJr;ZjC(~ zlUobB+S1#d`yqd6>uul8;mxoJuX|FZXg78Rn|i*f-KmBFX33X)=0*Zr9(NFmH?0_g4E6%icj2Og6w{ zHfs85j?c$w#K!kgFUoo+eQU&P_!N#4Szva*x(QwQLs$!Vy!=&E;%6QqyHmU(Gcq}0 z*=NV{75P9O0Ub17NY@2{Yg)jo&*#X!+Zr>T>Xft6^|vfDF;#tNTgfiTu<|_FIlcEO z8?)oApgiOkceihr$PJ1CpZbhQc|%7I;g?7Z2F=8^IaVe! z9-bPZRK6o1QO>O?MxB@67mhkGtRG-=1-j;S85_-aA;?{{f0v*uv| zf|YRR7=GIqdaEkYwnrsnW$Q7XH&f4{kS_QQ`bmh~!YvV*(1#MwueIr0%NPDe4^-r+ zyv2%Cr=rvFdHI}4$rl?5wzK4f2UCs37-f?P#QJi=dpsCqADSF!Brw%l5-T#%yWtWw zlN+#w4H*CjQ(dmY5-7Ll)r@eVVw@mB0q79a6xX8d77yF(mQ-Qe+unMvMrBnBKz*6> za-4>Y7p!3DUz^^?XX{|ac$D)aHVEUsA#R_2vm5(urMYD2o*Qmv51702f7*F`u z8K~}<9-R^(3kF z-^+ge+`RlB@A;24n57<+j^6W8Id1G@`OK;wKc~nGuNP-0V^B{kR0KM0szg0tkCR#Y57RFfytn;5*SIXsdY=Gz*vfrb4A!hYY;GoL>Xo{82mjbweEwEzmu(H@OaqLr;=j+^MqnP zhQ}+Jxr6rOXzDvi7wHuRAK9~S=9#F7D5#GDH zV!5Hd422!j8V({T6HPFbTJ39%rBu#QsmkOL=w2R0*{&AGz1(*fferuT^DAw8hEA8} zjw6ADg!KmH&5SX_`Xwdca^1j3x-OPFzO8DW-=)HU6H)$_Thib-xKCk2??5}U(tdqH zcg&I&la*8WBqi+JXTH`R08l8~d0sDgZTr%jpZeCAD%zqfV;0yNb0!5EeqDaxwiOo_ zNqsa|j=S_{?A()QplYA6-2w(pZCmcfhmPooT?w0lfc%;7kycvc`OMd%cQUn_BIuCs z)6x}>J6N26E|s~FaPgj($*}MF=rp10&M&xSj$5(o3#r+S5gCiP0r)NZ^IpCTXlL#=WT=M+REyXOV5*miqCY)T3sHL6thY>eopD6W?Jfqd|yEADM8?v%fz7KFUw7i^hM9Dp?=XubMh5WsES1aQ0?u?Y2`!O~y0f);qZ^YZ)u8&}FZ2VJxy=+t}q>JPe-L1g!F5>YVI)vl`cIU+ZoS-nx|A>lo;C zY3nQOXV(`>OGLIgl0-0tIa4*5H_u@oP}3;Q{@4)V@#>Jb=iX(k=g)LpDvZg}rRM4O zGfg~8R@H|dyLDVpK!5#oI)%kAc$%R{rmjMa$~q=Cyl5*eGA@>FrqB5Js#u1!11*T) zfgnS|y}znI4R>g$K2-`Hm5t2`(Hftm{Yk{I6RE>%lpoIZ(zgOj(k3{R>kroY_&3HkOhi+D4vbFI)pLK zCaiFNOlRp$O3>n1rLWZN_0alH@JsYl@82dt-^s}TUt}; zJGtOM?No|jxtCu~+O_ytc@1m%`Qk&mLbi7DW?-t5$LdP7<+82v2ZK6g+ZPe40@3G{ zGs-b~sRLsgCNm4Yb~Rfy+wtr9*pUI(MX*8)AiM;D7=zx7c=>)}SI5BBBzd`mY&muG zn`^meN>_UL4SM2xPx%8NZJkO*k5c43%#>6OD$?!o`RJ7J+&NMzcuHb}QMn|_HM&c% z5}?2LJd?!pQH%?;NC+MemE%&e9c8hDHc8bO`lmmuog#Wof%{-}lTpzR>7|6Qp?;mt zp|n>~QJ)JSUFUzfk02%z8pl8XVjIYYbdix&T5n%eIglYTmly51j2Mlk8RQcf<%DSI z7NM*f{)3y-IZ50dGCb_}?!1mxpnm&$^RX3S0Xu*>pydl!Rxnuw+b6}}E7M(fygnaR zs8Z~;HjZ0rSVS3+kkN_hU(gmVwOk6l;KA@kt6^RR?=DvWFx?V zftXx?P<*Tszdz9HNV0IX&cBoH3Y=-hHQLd4>S_i5MwCtY-lX`(^i9eE0N#rdIP*SrU$dW^CdwbGw zh&jevyNNFsNgV&Wid!<*H6&3hT-iaS5U{2WNVd{)4fG`FWor_R@4%}k-(M6`)8o+t zvsGeJ`FHv{B2OUtVCbKU+3oc9w_a%m2EtO-A$Dc!(S23~F{Ovbt47@K?%vK1ii}O@)^3*EKM~~1Z9i>$ISB0rTU0*3}N0%6< zoQ;nsq`iRmCC4jqM)N<0dRXb{9Ykv1mHBX${c9Nv{kdOCgRt~*f=Qo1ZlF*$QCYH4 z(q;ifh@q~y5hE&zO@^3wLxcE5?n;R3_=c1b%a8MIfONqOE~~m6X8CUZkgkY=PTMS3 zmUnmbkt?E{4PeST+v5X3IGXiY)S;t*%>)aUhssym$cj$^0JO*dUFB_d1mmJ}@a7X1 z?{CZJ0ps#shFnG6n%;tp_(i8McMyAtb8_Jv-+s$KKW)mXsYCWPOFdv%t#i(?LT+x> z=l+#Hk+2BsRY*m6l7`92y#xtI9w{3cBqt|Vx0gWo9{!dvnyW8e(% z;Z@{n8A$`%Q@4Z-Ydh(4me^27v);&<5IUM81X)Z^3EoW-F7S!9d) zZw}DS`A-{1`QSbFxn1aiDL2Y||92qAW!x}tEAPSX%g)k94ssgxN`5wf?#9 zvdCFbjB)fiX_SU{4OsZ-*RBP6d2T^`a5zq*37s|(yGW~g-HEo1;Pa0}R#e(`X^|Uk z$sMYHvL3e|adUJJyqHxu3RG!dVvu))o^&!GUD;r*)91RPj@=nbn+abX$ft_24&{gv zc+JAt%xwwEsJKUP@-7HHVC3;!!7@0R1S@ zxLX3^UL1Z-5W*oGB|UgIdLlZyW`&SbusgDgDqAEZQ&Yg1r+!4eY29AdT!hFaj$RZf zER>It-oZInI<9^a(s4$|%KE4;IB?fxY|rS}ZEGU>Fr~?=rt2Bn4%aW632Z_` zhHvP|NB7GJAKXTC1xa{ylPWi+#vYn_|DEPP7re+6h z)%9n;uRyYKl5^b9rr1}cz(v{M-U0r1E@||_C4HHp_YplrLQAsoRvui0e5w9R0-u<0$_w>mic)@-|& zpSRH3;GIO$;7)wBvhsnxP*}P=@s^^BE4>hMMMP8bD0MxXLL1kzrfs4Y)`h5>GiveC zKmGSzQT|BLj(Y5?Jn-aA67w?4ghsQ1-+ytZUvtvECq9wfs%>UB;gCnwl+^HEsW@G% z^L!#9hc;=~n>9Xjq%t|Iw+$YZwS0{D?if7*Q??Ojrb*mrACRd;abJXBw!xQJFxZkS zLzMTJJT$sCQ-KzIcACdH=OxMX$8B66S9HIcps(Yxz*GhEaccqdUtCT*K|x8MYu;lX zz7@tkcr3njQR(tr#u!Yc>%;oOGF{VxhQwDEmwoYjF~nMe?nfy{U&oWLII8@vlR#;? zHz@TMx6hbTf3+x<7`lFzX$EEzCDmM2kXgF{`Ls`P`!M52iHBi5ACMg3zz@jH7ZL%u z;1wN>NMwh{71OM)Wy3!h_i756a+o76_81@N-t;eof2E>=9{g5q%A4F&AKR{b?9Jk5 z4}^j}9z)q>6~wlJhQI5uq&(F!CaMMxPu>kNFXNptCPDn2^(it4h_j zpAuX<8r8@Sp|_w;)TYIskm}8u+67vCsiJrQ?q2B+47e0IqD{O-9lwS&P@M}!s-0j& z5N2ME=B^&b$F{nwNa}yKv6f;)!2BK6 zP0QaY<&z;u9(YexG`d1MO8TrEm8Sf`oEffk>v^=a`eQhKY0kZJ%pG=GuybXGd50qfkFhi$=f5)B|GRDW=ac^)nB_<8c^v;OihrE1|6hvY`|8jw zpM_KZSXXnC(@z#HW$!F%&y==gB3T5SJ*RU|Syb#K0)_rZfxF!fVXe6=#tSSnLw8W?$gIv1e)_kRl>B%|sWEsM zU@_`w+1eL_m$%+%ZcP`s!1DeJr>;Ee54{7_n6X*B|54smTy1jhe#9S|oaOH%3WUHo z2f5p+jV2l3LW730w)PnwOkvSyBiWDQv3aJF87!(%RV+GyJQjpaHh+u>5z#Gp8ds!y z4tMp6vHIru{9U&$_I(4>mj$v@oD$X(W0Tr87f6x!wj z^A-N|)LBP9QRu8@8Z3;Ym4o!_0AG{Ti100r*SFqKf6aYGDIK?Yf1)3I<3txvY}Rb| z!0Zy>k;m*nR!$L1UkNVV3E_?nFB@*lu?Q)}y}qOooz<4I(zetE7P?DK-ypk!?1Z4vl=-|b}KCsM!l=e9H z4BHh?7AKca!tB-jfR89duCdAuMrP3sGqu|+YPx2=ViNtD-^Z~fti`k??S`3MDANUe zwy_NC%wCj_$_j56o^}ji9}(WLN=dU)5rIamipaD{yVAx$>1X@g!<-w>mu!cZxwB@d z2*1e%qR;oE%I4nF&lTph#<^fPETMqOpM!!~Wq;mt-1qTG=2uz8dB{Y5?Xr^Fd?D4e z)dD3_($BA|y*HjV(aq})Zk2M*D-u!!9p7z~u%X`_Y?0aFGK$Si1qjl!vf5&^=B2i3 z?&xM{UZ^_=xN!(VXfn!va@EAryuVpcnxOaO!Ebb4JG++BSU_u%KTc;IEeNg-(y)v(yq)F!sOE>h|j+0ywy zCg<)oweDYu;fFptq53a^-G5N${^8L5Q~7@jnNE@igjf|tTw$Tv9#PjvlC>89*Ur#l eg78d^MYgf1;rw(f)*q$-#s(HwOZ9F)`QHF}_P@gb diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png new file mode 100644 index 0000000000000000000000000000000000000000..b0911374843bc9a0075f796fac85cc5f2e7270e3 GIT binary patch literal 14975 zcmeHuXH=8lw`M{w(xi$MQ7MWv1r03`L=jPnNG}0F5fKOmrG`WiQ4mlOL|O!-C`tgO zmqZXEDnUUEolrv$HGzbL3HtlDJL|4lGxy$^4|6|o$lK0)+J4S{p1t3PHdf|B0vZjDrEZ@lZM=;eR`^FnRdjv!lyQM(uz5bWzAvN1pb_9 ze>y3ZX1FdQn|LPw$F&a{l|x7t8EK{a=h*eS0ci`5+=}z?YX7T_!RDJqtYykJixLt0 zSFP5`ll|(=Jms@#%zf8A3t$&e*-DY0$ zhcG7UyMO$65~#?mX|Vnoz4c_6R*&9#+kglx?X+h~4s;e;k8Cfl&1|a&u$#HlIz}Tu z2+J6vtetDv>RTH_hzEu+mh7)U%C`o&vx43*+EI4>O$}RDg-={JQX##CB>mNb&2)Bm zlaZUxkw4Ubblp$Q7lCAeyOhZJGx)w}NY%hRRk07Xx_rV5i*8JBnxlqqr_2Po<2H@- zp%EU)ZwWJd8=09i;(KbQrU6lYQ>~GUaUJvSCt?>CXU?<)W zzD7RlPll9zB$+RxTfpHY*adEp_sP3X1}88o7Ba2P_{0`s6gvmL0#$`il)lg|q=$HS z_RCdWf_-T_`Ma|VAFGF|N1_dx)M{+@&#uZ57Ik&zlsp$W3+yUa`R1pcPp+ zrOjQ4}BSgQ0y(dClh| zvLY8&DVeb4QQzD3Agd;+e#dTnyIn>x63b-HjAMH0_B!VFm1pYIFfX_&aQ<0hal^N_ zw>r0t3Kq*bpn#lNOVO9lO%_h01a zixhUQEkm+3LpE%HWXqwE=f}^=G23H>_p%@G1wSr^iDg&UY#kNQU0)X|RJ5lrdo3^^ z6_|RPHCD)ScNFrs;jqo51-keX63v{0&Pi6TXLDG}6n*?496wMfHY7n))2TrGzLq#O%L=8cG#q;Q% zs!pJJBUOB$$bV%k@SEb+6{DnL>pymzdA%Mq3V?boB=oaeII#YrDG$^2d~&eVHuG9BOrO$gr{b z?`RUcgC=yk3#RO^)$SbVRcp8XXHw1#TpVDoJjZ7JHrMWxsFbMGmh#q zwsCB=K(=E34ZfKCG4iT{sz?TPVQmM)Qe5iV4CdaxC+rlg$HdUCMN+Z%_DtURiUeWM+Q)n!(D#jcDcD%oQJl@bCa) zTTLSTLtTU>rE@UH!|H(fo5j{?_}i7mn+Ey@FEmG9ORnncN9I=#qGs{y`UwOeK!-zE z*4_HKb;{T~S3do^UIHd@mtu8rl7qpZQSt#azfrj8{A4swmd8^RCqR%7bS+=7*X=Dr zN9c|@GqX-mp=p!(veh+qEb`MT^J6)oZKs$WgX+fWm{S`Wr`96F|D!}j;Y{3srjgCjXq&yrd zWq5#)Rqkj*tVnD5Chh1wY%x|XDjjhFG9}yWd*&^s&O_{)0GW^3Hn;fxVv`iW)eL8H z%n@)ZLGZr%;iltCd4;hB&WtI$-kjr9SC8Dd1C59_+K=TR^ z(~&anuCjp35l3u4CNn|?Fl*@|h!qK1wNcg9pe;%5o=78xwd<{-V|3%#F%^Os#rGk6 zazJpLx9kv?+~5OEIe4y`WN>wdu%rYflHW8u z|Jb!&NGWjcr1YH+QY8VBC4s!zHJ)Mu7i7U*riWeC;#VVbMSJ2eM+mnz-li79=9}O( z3k6k2wT~pAzt4+^O%47qUi0ou_)vS>{^x4I>!s)#oTMV00mRQ&M>TTE0p>(u-`k>x zBqUQO^c+tjo_uS{JGTK7i%-{T_FJx~fU8-YJrq=u-6GX@UChI8v_!`F(Q5M%j>oeH z_L#dBukP2V&YU}({4rZqb14{^;H{qJ%9AwRd2=L|QLz}&!cgh_*WuDV%H15x*XQ;_ zoBLN|slbIv8ON}vp8H+jiB6KgW9}DORO~yfPRJFEtTLYQ)iS*QVfsw$NVesg2Iomx zp0&F9V_IFoO4Z(K_fPF{Xs(-uG>N2^_rbr;C2-hkkbKCJ_;+<4Iyueyrr2czAa7U{ zAE*XschHpYtl<5@9ErC1^%sJQa3z;j;-|ze{Tn9l5vR>DM&`0NT?D=QtwS1**$sAn z(tE%GPDpYj<|RZQ0?9ZlRTEw1u=Uba*^Li5f2{P**OC{0Y|Q_q21Y97&|hzbm~HUI7AFdVoI$f;Nebs5JbprEIVtP~0&&bUxR1;q8c{VE*9q*9 zD_2$z8-o;%atJRbD~z@Jb3tSu zfTYbOM-mo;0X%nc2*cvo@6l#o6B+CYxgDY_-?1>Ch_I;7(L}!?S{Hr|r4~d!V8e~K zC~OHAi-L!Q9LMO3=#bp3$=0{K1Eu?C{OeyE!b`^!{OhB6JKUBK+*ivvz^tk&FASlt>Nwf{Ymzjwb`utu6)Cs)TVXkG4Cc+1dfec_eCy^IFS= zOOK%!idswwJe-7q)tRtRqin2R3A@R`2S}tlmK;eUx)!+bQ2lH(M&A^NtyJ`l$cU$M z(j0bjf$U%U=K$oL?T7dOrxc}_A1p)X|8G+ua5WK-VFJ9zHh zim-V*8L8If7Ujd=5eWmp_yCDmG(w?8Fx^Az2-WKD_Bb9+vq|dmWET3C+3Ln0fBpRKJg}}%)DBYOcm6vR0YX%SWSCl`$b+x z^^*h zw0G*9H2H+E_sQS~E$VHDDANrr-(rM7f%O~lZ(5raPl|`!Sk{mjo_=^=^Cw)1xDY0+ z#o)B6f<6?|semF0tvY?8%8XwO>ASHT@4+6?u?IEWy^p5pUOPQroaL(SCXR}WL&flF zitK;$vKI5E3zbqig%bZY?y#AS7TD%2j=%A``0JQ|Oj}p?Q=3mI>mMF`U5FHjpsyWp zf$FG)-_UdV$&%}XC-$IHpe^z^1(1P?3OK zWa$rzuS33#_&QcwSb2sQ*J6ePF4}ByPIH)C8})1bCNM}7qH^p!W13dNq<=QOo;~h82 zyI+=l7obf4|FrZ(X*F$JrmuBd`?1;AuiDp}w}d!^wNZ2>F2Gv#?Pz5ZfK&8ep@saT zE`yMiRG4pS80j54GeMtxxJa9rtVB$D-6Z~4WgXcDG-X(Y*87nMX13y&=(}A zgcR$_P&ToeM)~wY;Ke1(R}9Yd^FwN? zl4UlfeI~>p?no^JFnC{ai^S^{SaG4z;S?_X!z`OB%G`$7){JAWn~E-t;<%H}c?N&P z@ur7s(I!sQMmZ7@Z&J8(o{4~{E@r5s-fz~lFpWp2U_%y)k(93nt%|rxYx`e3a`NZL z%Er+g%H;SM+nI+y;z`_qn9oHZOY0yBb_Xkul78k?eS0T`y5yzng9epvdZ+bY5wsLFJ8E$y>8g#*Gwid9cAa)#Bovdl|%!zwUARPJ5GWIj$!Sg&`N zs*0x3J=AD(?DPSC(O?-1IayeRf0lilT&ZCr0YSTx-7n%P>y$OUeu^XMWp(Y#3JQM` z+kF4IQgw;>iz#=SV`FM+F^^`MHQ+9k9fB?v8!H+ih0VDrECjl2902yza3kJSjBLv& zW&vhNV%vJb;@Y1a9rwe`a$9*MrD5^b@WF_cTTXQtVXMGw6ekI&L^5(=2-ZB4S~cQZ zqx+AUup|od*f_MTCwbRGw+)6^2Y;ajwy%hgl(uuR6G3L~4@C2;-n`sjUH@2@lH2lq zt1j}UI7OXcfdej~L}9Q~;px)_GLMd5{oDRN_`Mi4-jruBua1tp|7ZJ#o#Nz3Vtnx% z-8+V1-~@?Qe`ZFhZDeY^#q5B!p7i znzXa(2{}Z^+0}@;sEDKPP47$c8}u`ia^l_(vzTsie|y*uXWQB`U8BRL!l5} zUOCCYpOp_#aH!6om*K~Efhd|q;`VB24?=-6Kihdoo_nqu!JL>Z28W2MZg zQ7Y8hE#dB$oM4B=TZgpo>y)H5_>MJVB^a&Vx4nZGmPl<}+%F|p%@vc^_^s4DD=%lo z3XI%z&4C$vX>?S|iO}=I1$dL#5|_ReaUM>CXTP(yVS7s5i+$?#_gmcmICa&TNm^a#=Z@VoTXp?c z`*%Q@8fB0+fX*5~=bBHg&s@c1qq_0=O!=$^(YEchMNUGqUCx8Bqc3T+SlvZpHZhKO z1R1L_-}+tK@E~a;xW3G@bAb{oT)&(kixCLxz;8P?fw@2ltos-e=s1Kjl&Qbh`h%y zeN6TbL!cwlurB7i>FBh!7j9p3P$?sV$}{;(O|K}osI_N@TlDIzY}&EqjIpAG>?8LJ zVPEdPj=#bjS!=Po7gj%OE@fMu12!(xINWvHt+f4eKzwzS=@X-DwTle275VU`8P?$P zic?>aOtLH`%AHIHe~U++X=R)s0L5)zzAsW(cJkL`FfyRpZ?q=QN=hq$w@p<6ht2GQ^^ zTyS$_feUlyk_DFk+=ZzMwS#jJ`%ldInLJ>1Af+$^FuN#$e1a`MC{5F)z`Hfs|x4taiBUp54JOoh#M%cg@(j1TE2R2aR z1hHoGCP@GT~e`yU&JwH{Qz_@p1fM;0MVdRHi^)DdcD;3{5c z&;!^ssafZnP$_%~^_KTDq({eK<_F|zbia;SD+sCBwX9h(1XX4>JOHojPaN-#I%%^7 zUUvG)v`kiS#3;`+b`B?;)9$0~aTC@Rq>W1LbaoBvlXcv2A`14|W%rS;Dx0&Zp!4%0 zJUy#EcT@txaTb~Cy-bdzfc>a2aMZz$5j1oB5vFJGeaL*EaR95I%9GU(TAb!j%6KGb zV%jB6XEXG>_F4*GV|6r)!j%L}4pd=kj_}-n17pEFbP5Y#N~PWGUO3d4wWJEStsyOCH+WOPCSu-ahGh_e-c&e<7QeNfw|r*N8&|8y2SON)VTwyDOs5h^N48ue z7cdX;r`_fK<4cBo?Qt(^ndfqGUrenEtb43HJor4eZE%)0URN5I`K?i;yhAGFt95~VUTue+MTTMtW^A2 z_8iwb*rmFuzo&R$0`iqM7(h6RUJSM?F2m0n$$1)*K1b5UuW zgl{|_VIHpEsa#p}3d*rT+hN_2O=nn5#ZtqWbVlAM+KCac*#fJ-{7dAkn1_54SB*bx zL>JQb%SmU$o7x;#ZxvBa|EvxZNdw1lW#MB`b|$#m07ou4qmzX=M|2rb==Z*NNfZ{i zm(8S;qc6fb(LAJ4FV-~M+24iZstjiNpIS~8_|DJXBL3HVC7THW&f5T7bdYOo7l5bt z3)dtoa{rdzH4;@6_Irc9arti7w#kR?nKd?buerdL1>Sk4fd4Q{tL_WdqhSr&Z9kAy zv`Sn?_UPS6-Y%&w*;X%)d>!3`1peBoG*ARsB*JC|Ck;$yV0K#~%`X5?m_>`-+e3FD z4m`r^zMJS=>t8+M=`k#6jytk3ds2)PQ3EG&Gdmp;wYFLGd*Gf}9MGPUI=!9xUR1kc z8}isM8j5A&wq@KU9hgs6CAyCpxH8z<=|;lc?>VR^ zhzpYRTgcY>L5d4od#GZY=)dE8{G0Lgyvy-Delvfk`v7qXl*2bggeVLJOn(XZaQSaD z_GdT$-=3=fuKK^n`rkOd|MdeRygnRBnNj`+w8X(i&-@*GxIH!Y1WK?y@V^@J>ugJ6 z^WS}Q9N4dBG!#bP|Ho^0x+r);;2#y~S^jfHFMXgjw*Rw-fAWIO4h!htVFCXU-QwSS z))OVI@kmd)5CbH27rWjoVB$fW0!a=vjq`Nau1mztU*kKxM-ln78ac3tg4rYE- z+%oyaS`yiY(Pv9A^B&9{6W`Ht--ifavJN~q+Ez(HP$F-4M)QFlmXOW6e>}Oa+C6E= zxa{cI{bF(Z2FuQmIaLbHIj^dPS#a5A!P#M+pX(u;x((*D89fZC(t430%4!QHe&@B8 zbPGm>bH2+=o#oSu$al%5T0`m3t<}Q{+>h<8| zVWiOOF8!|HsZmvhpswxAW91hVpiM08tIJ7!kH=_eZ|TlWWo zAC+uQn@VWgf1hW~_Kt_UI;aIWfew<6Na?wn_0*2B&I1z?Zq-7>@6F$mj<4v?asMh5 zPebeiEgS1#hH5rK8_nB&7utmRKqqR12NjJuldfxEADzc7PuKxBee0d!r*(pQ9*<_K zKlPI-ey7?YH4I9xx!P8Hob-$V${@^>iz0-aN$@WCgwm%EFvwSO*+94FSVj{sn~S{@ zK?? zLH3)~)2-*^{XN+HWsG=XRFKizOgCt9Km$^bgW;=+%*s38MU32%E zpr?a&rHzEU%#yY2({WQJA#P_Wf(vdL&rs!`Vd+RabisDFWy?_nvC}~#de4#`9weWo z+l|fLWRUNuHjOx}41bIC1?n0P4?X`5T|n^BG$DoATcf{Py*01bX-Fj2e0~*+j6TgJ z``%arcx!&SwIsp`#(h9g;0>uR`uZKSG9cN$KiLJ1SAe!;pvK!4BQ6eHDtmyjhOAo% zmem(k$n-DIyu8vWme{U9?ktKqpR2eRM(7D&pn|^lMGuwVyb|@e-WM&(oiGs&q+YDX3(=(_e#vNWRH*!_i^<%Us4an|L`w`WsEJ-mw{)? zxkuIl7)xGC9@dtwAaeDwJr2pg%zScG*HB*J*Ur1QmK zbr>{QG0(fBX=V5f**1P$f{0-xj4;;dV>A=zzBtLn(A~SY7(t(CCil|k5uZ7&YE*9z zWkc$tKT_v9{fNXzlGXlDoGsQd-^1>m#I>}35#+|Hfw`j`Nm8tZrk|#{O@}22h*8g9 zI;2+v-RlTok=rK*k6dUJ9A{CjxW!d;bhussR8e;1p@7#pNi`kk5=6?~3BiQyEkv({ z!2OXEoA34UIpy(rV(~8xN-sHGK>F!)mfEhpZkxxCW6EI*wxheptxD~Gct;$iWe&L- zf`b(s*W_%m6;69?b32F4Yq2sV)fgNe|B2>QhL``^B&VmMX!Z=WYl4zlf7`HX+DR2F z;&EH(=Ey{ZWH?rkl3P^WO0!NHLQbVX#;F?IaD<+o;{x%=pth11(7-$^cnNk)I9GpUx*k$XJw@=$HB}(n( z8&G=FT5Z?e(S+>&B9F5PA<#n6O^*HXz8-2AMSp@MYS7Q2avZ+oU-m@tBFHcrF)5cehP(8}n0Y>75}@QA4Eu3c`BAB?oY zpinDh)`&x`lyW_f{$jJ}{o8{Hjl&;$PIcV@ z;wk7VqsPyx94TH(y0Z=fr&*1b8I%K9trm_XPk{?=TX!Qq+TdgFKIQ=)d-3(MB(pi# zLE>ll)cTrqxHN%sC030UJ6>!f#W;pJ-5PK;{z<`)HU)Ubk>hh$*7@)8n?6wtqy*ry$zUmCyDh^m*_dwqmaU=qhtkj2dko)b8n+) z&*Ck|&rF$#@;99qw^u!EKR?7Ur-0@br>vrH$ogz`XH7V**X_Z`6g7wxpz>cSnKxLs z>WJ`x5@OF-%^qtE3cl!&QkA~ca}X_|uGh{m&1XoaMCW===k25YpmG6D(TmGu^tcYqjG4MzlUy@0-PBXsv#w4Fufog{OI0Zs>VmeKGL~P_AReLk zv_%V-rLW9Q@2BOPDtHeH$ffJR1!zUCwI<9h@IT6;>(z4pt;J(sF^p>b(c_U)_m z$kOjKDy8pW@0Qb_UcOF=@eeLOi+7wR%BzovM80M0HG2~qX;X{0&`APIcM@8CKR$q* zSZNgV{yBb@l4z(>OI!pSZ4R$tyBOJWj`aD>fJK7}#+AkTU7)E$-Z!(Rz>3t2p5CX? z*67%|PVlkY4SjPp_60?~vq7e>i?j=tDlo5$z;XLYaue~W82fXOL0Y^fM=S}IY1tyS z`{=x!nxK{PVY9^n-u%=lTpFg3HJ!}5wmy<^P7Ee&iu-6lGbT-|0@|?;^Tjy`3Nf5X zQDZTNaK)*UZ0d$%3V{}vA>3gy>`UVuNt<<#icZjgK;GVB28)XqaE9&D?mf*`-_yll zsKkqqLAH}hO&@S)NA3Zr4yFIQC1*p}@?CaxFJo_D(r-oJf59(bu$AlIS(d&*^9ioh zsT&3n(N`Sl|Dcz_o~rw+)KCWFU{5>ff2H=qmU24TEka@{Mzr+dXjcXu=fY zyl4fZp+3uMt-G0keSG&e`5QD=l$5h$ZtZ~vhyX{$Cvyqoezdny zj%khTNFly^)3rW=GSn5>u%TbC9I>n3uc$u-ehj&5X|^xWph>1DQvSC9vH$43i94Sj zaJCMHc%D7=QO+#qI8WAx4(0~wl`EfIpVK8cQ?hL$L*bMT>7aF4Ucj8Zu1Ol{-dkHO^NygZ){XR+j zqb+Y$@ttEuXV!H#ica?F)xqVYu3cOZzclx$AI-< zBx&lDa(PN?RO9W)t%>GAY2@2!yX@7CvD99VX~*F>AEOncnK~@bKywv{l-_xM__?9x zBgf$rKL(9c&Y_PnGUi?HA2}i1{j~G;O^pglBhBPclvKX|+J&4G;Uir-e%BQ@bA~j3 zk87Bw(X))L#^-@RJohso5Wn_B5vxK1Ei*y$G*}l~(yp`IzUrhE!3E!B4-n$UJ zh@Nt~Tsz_-Mw$C)Su9q&t=7@~;3Oi|`zKj~)|zZ$$$8pHMgO+43(;E*d1f$u(AZiK z8*zb}S*peDn7^X`9?c6;=}V@+4;$&abt_=fzYngMHJhH&S@taHwND3_^S88wa}p4I z^RP(9(o1EY+Ot%@h6qcSa-75EJ?f9~bi&K7?7a>y;@^vJ#97$@s zbol;EW6E6Mjk`5k^sJReDcXTKN2sL}^zghhR!kiB1&TY&YM z?|f4AJ85I8EkhA11Y*@}88jb0j&1*V--hN#8!%?W@EH>Uv=}g4D7b%e}^Ue!Bv_X^Loe?Bu+kw!pg( zXxb5Lo7l1G7ZxWp1nSCCRpwjUYaD)l!IZ#Xai3;3!yIf%PG5mygMa7{Cl+J$9kQ*1 z(4E0X+kNhtvI%nLe#Yw2X}h>*f?sbaygGfcv}E}UylrkJ>zq;3E2#Xh<_O(0t{cqq z_9mTUJga@}5c9f`g64OO(kRX6VBen3m`0JF&WLk~`t$W+idn6lV|B0sLdtNF9zN0R zcXz;BDUT<$SDZeHKYMXw9h=*qCaHjZ=hkpO_jC1Y@AnoX&RJWlOl)gL=If3^V2$8YADUvehQYW%}t2X+E(glq2>umr|f+zh!A^Ds z;HKDlY5%Du3fQzUmN3(84Jp`p54muPsJFA6X5&Nw-$(j4Q`t>Fi36f^*eB)LgaIeR zmBl-L-N4WwBk2-5Rn>jKQDo&W@7oROyNXtjxh22o@m6Mg{i}E*rQH%vEX~RE?i`-( zF|6F;+_-Kqv{0BGRJ5vo;7|JP^6Bm(0l=a%2jP^MGZzV{hF2v8M=THWWy{@CDxEhJI_lK9|Zk-2ap5JA-D9F+IrrS+SLa1!ne-5viEdJn}pC$;sM=1{q?xA;&9TvwWKeQfsQk~ z1KsR^o>y=W6K-8AUGi0Gu8hTeqf?#vL4E5lrcBRPqsBA&JzM=(Q-b&Q(+}e${X3OU%Ws%KkV00F-3(QjFrg z@b=^DEDeV6RW5=8UwaynguLO`JuJ_PZuwfRj16 z)j1*}Ulg|3*zmH5+qFF*i2xzB&1gZ+@^73TCl4(o4+s#GgmV=g^E`#BF!6L#34$#kI{L^?OkXpjKK!K6BwJd?Y z3hs1C`-GiDxr6LPIo>5U^Np~x4x}14+VX*D8G=fiL>)gd5r#@fK8&*t+@=z z&_7S|?-x5{EYH?oLb3#T-lTFS?Ur7sajNE$D+oe|;~rvM8x?5fTBLj6 zp}8uo@o)ER8lKHwBjTdXMB~_^1IPJZEbLF8=RZ6O>LJrn;>mu)l*_{IV3JyU`euXx zPM%HBL$CcUSMGn#h5x5iT6VUp>9%MW`}?AIL3XQi|H16pe=iEfKCjcgwhfFdasF}s U^L#4%qfvmxX)Ci*Q_s8q1CS0Pz5oCK literal 0 HcmV?d00001 From 551fe0ad183f8016ab67e94ec0f86d0dd510eda8 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 23 Jun 2019 07:28:18 +0000 Subject: [PATCH 09/40] modifications --- .../launching_emr_cluster-1.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 5f063e15..6394e114 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,7 +3,10 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) +{{% notice note %}} +Normally our dataset on S3 would be located on the same region where we are going to run our EMR clusters. In this workshop, for educational purposes, it is fine if you are running EMR in a different region, and the Spark application will work against the dataset which is located in the N. Virginia region. +{{% /notice %}} To launch the cluster, follow these steps:\ From b71fb17d9e9fd58430fef5300010739548f69325 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 23 Jun 2019 13:39:33 +0000 Subject: [PATCH 10/40] modifications --- .../examining_cluster.md | 4 ++-- .../prerequisites_notes.md | 2 +- .../selecting_instance_types.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index d1b77a57..3f6ba041 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -6,8 +6,8 @@ weight: 95 In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. ### EMR Management Console -To get started, let's check that your EMR cluster and Spark application are running. -1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running. +To get started, let's check that your EMR cluster and Spark application are running.\ +1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running.\ 2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. {{% notice note %}} diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index a86c236a..d9b0c975 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -22,4 +22,4 @@ f. click **Next** and again **Next** in the next screen.\ g. Click **Create stack**.\ The stack creation should take under 2 minutes and the status of the stack will be **CREATE_COMPLETE**. -Congratulations! you completed the pre-requisites to start the workshop, you now have a VPC to run your EMR cluster in, and an S3 bucket for the Spark application code and the results. Continue to the next step to proceed in the workshop. \ No newline at end of file +Congratulations! you completed the prerequisites to start the workshop, you now have a VPC to run your EMR cluster in, and an S3 bucket for the Spark application code and the results. Continue to the next step to proceed in the workshop. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 2b734e88..2eca671e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -36,7 +36,7 @@ Instance types with sufficient Memory and vCPUs for our executor size, as well a **Previous generation instance types:**\ - r3.xlarge and larger\ - i2.xlarge and larger\ -you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take more time to complete.\ +you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take longer to complete.\ Also note that not all instance types exist in all regions. {{% /expand%}} From 1c67f3ee6fca19f48f20f008ec87244d3ad50a86 Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 24 Jun 2019 17:02:57 +0000 Subject: [PATCH 11/40] adding Limits warning in launching-2 --- .../launching_emr_cluster-2.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 0ab9130b..e4c7c332 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -25,6 +25,12 @@ Under the task node type, Click **Add / remove instance types to fleet** and sel Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **40 Spot units** in order to run 10 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 10 * r4.xlarge / r5.xlarge / i3.xlarge **or** 5 * r5.2xlarge / r4.2xlarge in EMR Task Instance Fleet. ![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png) +{{% notice warning %}} +If you are using a new AWS account, or an account where Spot Instances were never launched in, your ability to launch Spot Instances will be limited. To overcome this, please make sure you launch no more than 3 instances in the Task Instance Fleet. You can do this, for example, by only providing instances that count as 8 units, and specify 24 for Spot units.\ +If your Task Instance Fleet is stuck on provisioning, try lowering the number of requested instances further. +Your Spark application should still complete successfully, but it might take longer due to having less executors in the cluster. +{{% /notice %}} + click **Next** to continue to the next steps of launching your EMR cluster. From 460cba49164c10581cd2e5fce3292a6aa373f661 Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 25 Jun 2019 11:56:31 +0000 Subject: [PATCH 12/40] various mods --- .../emr_uniform_groups.md | 2 +- .../launching_emr_cluster-2.md | 8 +++++++- .../spot_savings_summary.md | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 55ceef30..cadb4a72 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -1,5 +1,5 @@ --- -title: "EMR Uniform Groups" +title: "EMR Uniform Instance Groups" weight: 20 --- diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index e4c7c332..d851fd3f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -5,7 +5,13 @@ weight: 70 Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC that you deployed using the CloudFormation template earlier in the workshop, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleetsnetwork.png) -Let's discuss the right setup for each of our node types:\ + + +### Setting up our EMR Master node, and Core / Task Instance Fleets +{{% notice note %}} +The workshop focuses on running Spot Instances across all the cluster node types for cost savings. If you want to dive deeper into when to use On-Demand and Spot in your EMR clusters, [click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html#emr-plan-spot-instances) +{{% /notice %}} + #### **Master node**: Unless your cluster is very short-lived and the runs are cost-driven, avoid running your Master node on a Spot Instance. We suggest this because a Spot interruption on the Master node terminates the entire cluster. \ For the purpose of this workshop, we will run the Master node on a Spot Instance as we simulate a relatively short lived job running on a transient cluster. There will not be business impact if the job fails due to a Spot interruption and later re-started.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 02b1bd40..a12e19d8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -3,7 +3,7 @@ title: "Spot savings summary" weight: 96 --- -When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. +When our cluster has finished bootstrapping and the Spark application is running or has already completed, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. 1. Open the AWS Management console and go to the EC2 Service in the region where you are running your EMR cluster. 2. On the left pane, click **Spot Requests**. From 880191d8ac81dbeac55aaf8be4576a5ab0bab0af Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 26 Jun 2019 09:22:02 +0000 Subject: [PATCH 13/40] various mods --- .../_index.md | 18 +++++++----------- .../emr_instance_fleets.md | 9 ++++++--- .../emr_uniform_groups.md | 16 ---------------- .../launching_emr_cluster-1.files/script.py | 6 ++++++ .../launching_emr_cluster-1.md | 9 ++++++--- .../prerequisites_notes.md | 3 ++- .../right_sizing_executors.md | 4 ++-- .../selecting_instance_types.md | 2 +- 8 files changed, 30 insertions(+), 37 deletions(-) delete mode 100644 content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 86cc426a..fc7a5ad3 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -9,18 +9,12 @@ pre: "⁃ " ## Overview -In this workshop you will assume the role of a data engineer, tasked with building a platform that will allow your organization to run data processing jobs, specifically Apache Spark applications. +Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\ -The requirements for the platform are: +The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $1.\ +The **learning objective** for the workshop is to become familiar with the best practices and tooling that are available to you for cost optimizing your EMR clusters running Spark applications, using Spot Instances. -1. Use a managed service - in order to avoid the heavy lifting of installing, maintaining and upgrading compute clusters that run Apache Hadoop framework software, mainly Spark. -2. Be secure - allow network level isolation and encryption at rest and in transit. -3. Be cost optimized - use Amazon EC2 Spot Instances, as well as easily run transient clusters (that will be spun up just to run a job and then spun down) where possible in order to cost optimize. -4. Decouple compute and storage - in order to allow to elastically scale your processing power independently from having to provision more storage for your clusters. -5. Be self-healing in order to decrease operations overhead - if a compute node fails, the cluster will automatically replace it and continue running the job. - - -## The decision is simple - ***Amazon EMR*** fulfills all the requirements. +## Recap - Amazon EMR and EC2 Spot Instances * [Amazon EMR] (https://aws.amazon.com/emr/) provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as [Apache Spark] (https://aws.amazon.com/emr/details/spark/), [HBase] (https://aws.amazon.com/emr/details/hbase/), [Presto] (https://aws.amazon.com/emr/details/presto/), and [Flink] (https://aws.amazon.com/blogs/big-data/use-apache-flink-on-amazon-emr/) in EMR, and interact with data in other AWS data stores such as Amazon S3 and Amazon DynamoDB. EMR Notebooks, based on the popular Jupyter Notebook, provide a development and collaboration environment for ad hoc querying and exploratory analysis. EMR securely and reliably handles a broad set of big data use cases, including log analysis, web indexing, data transformations (ETL), machine learning, financial analysis, scientific simulation, and bioinformatics. @@ -28,4 +22,6 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in details during this workshop. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as achieve our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back.\ +It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in detail during this workshop. + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index 76e64bb5..0104f3a9 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -3,13 +3,16 @@ title: "EMR Instance Fleets" weight: 30 --- -With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. +When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. + +With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones.\ {{% notice info %}} -[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets. +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets in the official documentation. {{% /notice %}} **When Amazon EMR launches the cluster, it looks across those subnets to find the instances and purchasing options you specify, and will select the Spot Instances with the lowest chance of getting interrupted, for the lowest cost.** -While a cluster is running, if Amazon EC2 reclaims a Spot Instance or if an instance fails, Amazon EMR tries to replace the instance with any of the instance types that you specify in your fleet. This makes it easier to regain capacity in case some of the instances get interrupted by EC2 when it needs the Spot capacity back. \ No newline at end of file +While a cluster is running, if Amazon EC2 reclaims a Spot Instance or if an instance fails, Amazon EMR tries to replace the instance with any of the instance types that you specify in your fleet. This makes it easier to regain capacity in case some of the instances get interrupted by EC2 when it needs the Spot capacity back.\ +These options do not exist within the default EMR configuration option "Uniform Instance Groups", hence we will be using EMR Instance Fleets only. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md deleted file mode 100644 index cadb4a72..00000000 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "EMR Uniform Instance Groups" -weight: 20 ---- - -When using the EMR console to create a cluster via the quick settings, default advanced settings, or with the AWS CLI - Amazon EMR will provision your EMR cluster with a configuration option called "Uniform Instance Groups". - -With the Uniform Groups configuration option, you select the Availability Zone in which you want to launch your EMR cluster, and one instance type per group (Master, Core, and optional multiple Task groups). - -{{% notice info %}} -[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to learn more about Master, Core and Task node types in EMR. -{{% /notice %}} - -When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. - -We will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py new file mode 100644 index 00000000..485a7c97 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py @@ -0,0 +1,6 @@ +import sys +from pyspark.sql import SparkSession +spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() +df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") +df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) +exit() \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 6394e114..4e68c2cc 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,7 +3,7 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it at cluster creation time, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) {{% notice note %}} Normally our dataset on S3 would be located on the same region where we are going to run our EMR clusters. In this workshop, for educational purposes, it is fine if you are running EMR in a different region, and the Spark application will work against the dataset which is located in the N. Virginia region. {{% /notice %}} @@ -20,7 +20,7 @@ To launch the cluster, follow these steps:\ ``` --executor-memory 18G --executor-cores 4 ``` -* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** +* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file (or download it from the Attachment box) and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp script.py s3://\** ```python import sys @@ -30,7 +30,10 @@ df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) exit() ``` -Then add the location of the file under the **Application location** field, i.e: s3://\/\\ +{{%attachments style="orange" /%}} + + +Then add the location of the file under the **Application location** field, i.e: s3://\/script.py * **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ * **Action on failure**: Leave this on *Continue* and click **Add** to save the step. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index d9b0c975..a0e922cd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -11,7 +11,8 @@ weight: 10 #### Preparation steps:\ -1. Create an S3 bucket for your Spark application code (which will be provided later) and the EMR application's results. Using the AWS CLI, run: **aws s3 mb s3://\** or create a new bucket using the AWS Management Console. +1. Create an S3 bucket for your Spark application code (which will be provided later) and the Spark application's results.\ +Using the AWS CLI, run: **aws s3 mb s3://\** or create a new bucket using the AWS Management Console. 2. Deploy a new VPC that will be used to run your EMR cluster in the workshop.\ a. Open the ["Modular and Scalable VPC Architecture Quick stage page"] (https://aws.amazon.com/quickstart/architecture/vpc/) and go to the "How to deploy" tab, Click the ["Launch the Quick Start"] (https://fwd.aws/mm853) link.\ b. Select your desired region to run the workshop from the top right corner of the AWS Management Console and click **Next**.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 9465f7fa..34a8497c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -3,13 +3,13 @@ title: "Right sizing Spark executors" weight: 40 --- -Building towards the running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +Building towards running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important best practice that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. {{% notice note %}} **Remember!** you might be able to achieve greater utilization and performance optimization when using a single EC2 instance type, but when adopting Spot Instances, the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. {{% /notice %}} -Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. +Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of EC2 instance types. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4". However, there are some more limitations in place. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 2eca671e..a7b02b64 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -6,7 +6,7 @@ weight: 50 Let's use our newly acquired knowledge around Spark executor sizing in order to select the EC2 Instance Types that will be used in our EMR cluster.\ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to read more about the different node types. -We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, +We determined that in order to be flexible and allow running on multiple instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ From f5be09abd0529b72a33f982d5be08aefdf71713b Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 26 Jun 2019 09:36:37 +0000 Subject: [PATCH 14/40] various mods --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index fc7a5ad3..fe57febf 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -21,7 +21,7 @@ The **learning objective** for the workshop is to become familiar with the best * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. -### About Spot Instances in Analytics workloads +## About Spot Instances in Analytics workloads The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as achieve our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back.\ It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in detail during this workshop. From 6fdebfa72fca4263cfe64b94d24e9641669fc9e3 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 09:24:37 +0000 Subject: [PATCH 15/40] various mods --- .../examining_cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index 3f6ba041..ec99296f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -23,7 +23,7 @@ Some notable metrics: * AppsRunning - you should see 1 since we only submitted one step to the cluster.\ * ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ -* MemoryAllocatedMB * MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ +* MemoryAllocatedMB & MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ ### Using Ganglia and YARN ResourceManager The recommended approach to connect to the web interfaces running on our EMR cluster is to use SSH tunneling. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) to learn more about connecting to EMR interfaces.\ From c4343c4f3c5fdc4d5c6f745c08e744d5bdf6ad89 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:09:14 +0000 Subject: [PATCH 16/40] various mods --- .../simulating_recovery.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md diff --git a/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md b/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md new file mode 100644 index 00000000..d6b58521 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md @@ -0,0 +1,30 @@ +--- +title: "(Optional) simulating recovery" +weight: 149 +--- + +EMR replenishes the target capacity if some EC2 Instances failed, were terminated/stopped, or received an EC2 Spot Interruption. + +In this optional step, you will re-run the cluster and the Spark application, and terminate some of the Task Fleet instances in order to observe the recovery capabilities of EMR and Spark, and check that the application still completes successfully. Since it is not possible to simulate an EC2 Spot Interruption in an EMR cluster, we will have to manually terminate EC2 instances to receive a similar effect. + +{{% notice note %}} +This is an optional step that will take approximately 20 minutes more than the original running time of the workshop. Feel free to skip this step and go directly to the **Conclusions and cleanup** step. +{{% /notice %}} + +#### Step objectives:\ +1. Observe that EMR replenishes the target capacity if some EC2 Instances failed, were terminated/stopped, or received an EC2 Spot Interruption. +2. Observe that the Spark application running in your EMR cluster still completes successfully, despite losing executors due to instance terminations. + +#### Re-launch your cluster and application +1. In the EMR console, select the cluster that you launched in this workshop, and click the **Clone** button. +2. In the popup dialog **"Would you like to include steps"** select **Yes** and click **Clone**. +3. EMR console duplicated all the cluster settings for you - click the **Create cluster** button. +4. Refresh the Summary tab in the EMR console until the status of the cluster is **Running step** and the Master, Core and Task fleets are all in the **Running** state. + +#### Manually terminate some of the EMR Task Fleet nodes +1. Go to the EC2 console, and identify the instances in your Task Fleet. You can do so by using the following filter: **Key=aws:elasticmapreduce:instance-group-role & Value=TASK**. If you have other EMR clusters running in the account/region, make sure you identify your own cluster by further filtering according to its Name tag. +2. Randomly select half of the instances that were filtered, and click Actions -> Instance State -> Terminate -> **Yes, Terminate** + +#### Verify that EMR replenished the capacity, and that the step completed successfully +1. Within 2-3 minutes, refresh the EC2 console as well as the Task Fleet in the EMR console under the Hardware tab, and you should see new Task Fleet instances created by EMR to replenish the capacity, after you terminated the previous instances. +2. In the EMR console, go to the Steps tab. refresh the tab until your application has reached the **Completed status**. Because some instances were terminated mid-run, the Step will still complete, but will take longer than you previously observed in the workshop, because Spark had to repeat some of the work. \ No newline at end of file From 4a05d788037b46fc1ce35f3a72ae2da940265c8f Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:14:00 +0000 Subject: [PATCH 17/40] various mods --- .../conclusions_and_cleanup.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index a4111f7e..2550ff92 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -11,7 +11,8 @@ weight: 150 2. Delete the VPC you deployed via CloudFormation, by going to the CloudFormation service in the AWS Management Console, selecting the VPC stack (default name is Quick-Start-VPC) and click the Delete option. Make sure that the deletion has completed successfully (this should take around 1 minute), the status of the stack will be DELETE_COMPLETE (the stack will move to the Deleted list of stacks). 3. Delete your S3 bucket from the AWS Management Console - choose the bucket from the list of buckets and hit the Delete button. This approach will also empty the bucket and delete all existing objects in the bucket. 4. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**. + #### Thank you -We hope this workshop was educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR in order to optimize your costs.\ +We hope you found this workshop educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR, in order to optimize your costs.\ If you have any feedback or questions, click the "**Feedback / Questions?**" link in the left pane to reach out to the authors of the workshop. \ No newline at end of file From 564ccb6c8557a9d945fd8b9847ded7fbb3b68842 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:16:43 +0000 Subject: [PATCH 18/40] various mods --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index fe57febf..c9c11658 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -11,7 +11,7 @@ pre: "⁃ " Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\ -The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $1.\ +The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $2.\ The **learning objective** for the workshop is to become familiar with the best practices and tooling that are available to you for cost optimizing your EMR clusters running Spark applications, using Spot Instances. ## Recap - Amazon EMR and EC2 Spot Instances From 08da026f4e3f2bfc9e674ef869ffc40b87a62bd3 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 16:53:33 +0000 Subject: [PATCH 19/40] various mods --- .../{visualizing_costs.md => analyzing_costs.md} | 11 ++++++----- .../examining_cluster.md | 14 +++++++------- .../fleet_config_options.md | 10 +++++----- .../tracking_spot_interruptions.md | 6 +++--- 4 files changed, 21 insertions(+), 20 deletions(-) rename content/running_spark_apps_with_emr_on_spot_instances/{visualizing_costs.md => analyzing_costs.md} (87%) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md similarity index 87% rename from content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md rename to content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md index 9c37530e..648ccfd7 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md @@ -1,5 +1,5 @@ --- -title: "Visualizing costs" +title: "Analyzing costs" weight: 145 --- @@ -22,15 +22,16 @@ If the Name tag Key was not enabled as a Cost Allocation Tag, you will not be ab Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ 2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ -3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end. -4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia). +3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end.\ +4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia).\ 5. Group by **Usage Type** to get a breakdown of the costs ![costexplorer](/images/running-emr-spark-apps-on-spot/costexplorer1.png) -* EU-SpotUsage:r5.xlarge: This was the instance that I ran in the EMR Task Instance fleet and accured the biggest cost ($0.17)\ +* EU-SpotUsage:r5.xlarge: This was the instance type that ran in the EMR Task Instance fleet and accrued the largest cost, since EMR launched 10 instances ($0.17)\ * EU-BoxUsage:r5.xlarge: The EMR costs. [Click here] (https://aws.amazon.com/emr/pricing/) to learn more about EMR pricing. ($0.06)\ * EU-EBS:VolumeUsage.gp2: EBS volumes that were attached to my EC2 Instances in the cluster - these got tagged automatically. ($0.03)\ * EU-SpotUsage:r5a.xlarge & EU-SpotUsage:m4.xlarge: EC2 Spot price for the other instances in my cluster (Master and Core) ($0.02 combined)\ -If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you Filter by Service=EMR and Group by Usage Type? \ No newline at end of file +If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you filter by **Purchase Option = Spot** & **Group by = Instance Type**? + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index ec99296f..d84d3ad1 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -3,7 +3,7 @@ title: "Examining the cluster" weight: 95 --- -In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. +In this section we will look at the utilization of our EC2 Spot Instances while the application is running, and examine how many Spark executors are running. ### EMR Management Console To get started, let's check that your EMR cluster and Spark application are running.\ @@ -11,7 +11,7 @@ To get started, let's check that your EMR cluster and Spark application are runn 2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. {{% notice note %}} -In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very resource intensive and is just used for demo purposes. {{% /notice %}} ### Using CloudWatch Metrics @@ -22,7 +22,7 @@ EMR emits several useful metrics to CloudWatch metrics. You can use the AWS Mana Some notable metrics: * AppsRunning - you should see 1 since we only submitted one step to the cluster.\ -* ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ +* ContainerAllocated - this represents the number of containers that are running on your cluster, on the Core and Task Instance Fleets. These would the be Spark executors and the Spark Driver.\ * MemoryAllocatedMB & MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ ### Using Ganglia and YARN ResourceManager @@ -34,15 +34,15 @@ Normally you would not run EMR in a public subnet and open TCP access to the mas {{% /notice %}} To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ -1. In your EMR cluster page, in the AWS Management Console, go to the Summary tab\ +1. In your EMR cluster page, in the AWS Management Console, go to the **Summary** tab\ 2. Click on the ID of the security under **Security groups for Master**\ 3. Check the Security Group with the name **ElasticMapReduce-master**\ -4. In the lower pane, click the **Inbound tab** and click the Edit button\ +4. In the lower pane, click the **Inbound tab** and click the **Edit**\ 5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ -6. Click **Save**. +6. Click **Save** {{% notice note %}} -While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which is not allowed for outbound traffic on every Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) +While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which might not allowed for outbound traffic on your Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) {{% /notice %}} Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 065696a8..b66b4927 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -3,19 +3,19 @@ title: "Fleet configuration options" weight: 85 --- -While our cluster is starting (7-8 minutes) and the job is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the step is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) #### Maximum Spot price -Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. +Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity nor does it decrease the chance of getting your Spot Instances interrupted when EC2 needs the capacity back. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units -This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 80, then 20 * r4.xlarge instances will be launched by EMR for our Core Instance Fleet. -If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. +This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of YARN VCores that the instance type has by default (this would typically equate to the number of EC2 vCPUs) - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 40, then 10 * r4.xlarge instances will be launched by EMR in the fleet. +If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the fleet. Since our executor size is 18 GB, one executor will run on this instance type. #### Defined duration This option will allow you run your EMR Instance Fleet on Spot Blocks, which are uninterrupted Spot Instances, available for 1-6 hours, at a lower discount compared to Spot Instances. #### Provisioning timeout -You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the Fleet when looking to use Spot Instances, before you look into failing over to On-Demand. \ No newline at end of file +You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the fleet when looking to use Spot Instances, before you look into failing over to On-Demand. Also, try to select instance types with lower interruption rates according to the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index b01eaa7e..44c16096 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -6,7 +6,7 @@ weight: 100 Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} -In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful. +In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful, as our organization can use these to correlate to possible EMR job failures or prolonged execution times, in case Spot Instances were interrupted during Spark run time. {{% /notice %}} @@ -32,7 +32,7 @@ aws sns subscribe --topic-arn --protocol email --notification-endpoi 1. On the right side of the console, click **Add Target**, scroll down and select **SNS topic** -> select your topic name, Your result should look like this: ![tags](/images/running-emr-spark-apps-on-spot/cloudwatcheventsrule.png) 1. Click **Configure Details** in the bottom right corner. -1. Give a name to your CloudWatch Events rule and click **Create rule**. +1. Provide a name to your CloudWatch Events rule and click **Create rule**. #### Verifying that the notification works @@ -50,4 +50,4 @@ The only way to simulate a Spot Interruption Notification is to use Spot Fleet. Go ahead and terminate the fleet request itself by checking the fleet, click actions -> **Cancel Spot request** -> **Confirm**. - +From now on, any EC2 Spot interruption in the account/region that you set this up in will alert you via email. Disable or delete the CloudWatch Event rule if you are not interested in the notifications. From 2b0ecd2d4c0ed20fcaa8485d1ee9ef0227536ae4 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 3 Jul 2019 11:29:21 +0000 Subject: [PATCH 20/40] final commit before merging --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index c9c11658..23f8c341 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -5,8 +5,6 @@ weight: 60 pre: "⁃ " --- -## This workshop is still under construction. ping ranshein@amazon.com if you have any concerns. - ## Overview Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\ From 24d581d6e4b41ec3174e9a293281db04abe16f82 Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 17 Jun 2019 15:51:13 +0000 Subject: [PATCH 21/40] committing emr into a new branch --- .../_index.md | 2 +- .../automations_monitoring.md | 14 +++++++------- .../fleet_config_options.md | 4 ++-- .../launching_emr_cluster-1.md | 6 +++--- .../launching_emr_cluster-2.md | 2 +- .../right_sizing_executors.md | 4 ++-- .../selecting_instance_types.md | 6 +++++- .../spot_savings_summary.md | 4 ++-- .../tracking_spot_interruptions.md | 4 ++-- .../verifying_results.md | 4 ++-- .../visualizing_costs.md | 16 +++++++++++++++- .../sparksubmit.png | Bin 14817 -> 0 bytes .../sparksubmitstep.png | Bin 0 -> 15091 bytes 13 files changed, 42 insertions(+), 24 deletions(-) delete mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmit.png create mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 3559fa2c..a353d30c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -29,4 +29,4 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools, as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md index cb855001..2f835496 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md @@ -12,17 +12,17 @@ In this section we will simply look at a CLI command that can be used to start a 1. In the AWS Management Console, under the EMR service, go to your cluster, and click the **AWS CLI export** button. 2. Find the --instance-fleets parameter, and copy the contents of the parameter including the brackets: ![cliexport](/images/running-emr-spark-apps-on-spot/cliexport.png) -3. Paste the data into a JSON validator like [JSON Lint] (https://jsonlint.com/) and vlaidate the JSON file. this will make it easy to see the Instance Fleets configuration we configured in the console, in a JSON format, that can be re-used when you launch your cluster programmatically. +3. Paste the data into a JSON validator like [JSON Lint] (https://jsonlint.com/) and validate the JSON file. this will make it easy to see the Instance Fleets configuration we configured in the console, in a JSON format, that can be re-used when you launch your cluster programmatically. #### (Optional) Set up CloudWatch Events for Cluster and/or Step failures Much like we set up a CloudWatch Event rule for EC2 Spot Interruptions to be sent to our email via an SNS notification, we can also set up rules to send out notifications or perform automations when an EMR cluster fails to start, or a Task on the cluster fails. This is useful for monitoring purposes. -In this example, let's set up a notification for when our EMR step failed. +In this example, let's set up a notification for when our EMR step failed.\ 1. In the AWS Management Console, go to Cloudwatch -> Events -> Rules and click **Create Rule**.\ -2. Under Service Name select EMR, and under Event Type select State Change.\ -3. Check **Specific detail type(s) and from the dropdown menu, select **EMR Step Status Change**\ -4. Check Specific states(s) and from the dropdown menu, select **FAILED**.\ +2. Under Service Name select EMR, and under Event Type select **State Change**.\ +3. Check **Specific detail type(s)** and from the dropdown menu, select **EMR Step Status Change**\ +4. Check **Specific states(s)** and from the dropdown menu, select **FAILED**.\ ![cwemrstep](/images/running-emr-spark-apps-on-spot/emrstatechangecwevent.png) -5. In the targets menu, click **Add target**, select SNS topic and from the dropdown menu, select the SNS topic you created and click **Configure details**.\ -6. Provide a name for the rule and click **Create rule**\ +5. In the targets menu, click **Add target**, select **SNS topic** and from the dropdown menu, select the SNS topic you created and click **Configure details**.\ +6. Provide a name for the rule and click **Create rule**.\ 7. You can test that the rule works by following the same steps to start a cluster, but providing a bad parameter when submitting the step, for example - a non existing location for the Spark application or results bucket. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 3fc0be60..df0384b8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -4,12 +4,12 @@ weight: 85 draft: true --- -While our cluster is starting (7-8 minutes) and the job is running (~5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) #### Maximum Spot price -Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot reuqests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the EC2 Spot console under **Pricing History**. +Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) are 64, then 16 * r4.xlarge instances will be launched by EMR for our Core fleet. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index d8fb0ad3..ff5972e3 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -4,7 +4,7 @@ weight: 60 draft: true --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the job we submit to it, and will run solely on Spot Instances. The job is a simple word count application that will run against a public data set of Amazon product reviews. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. To launch the cluster, follow these steps:\ @@ -13,7 +13,7 @@ To launch the cluster, follow these steps:\ 1. Click "**Go to advanced options**"\ 1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark**.\ 1. Under "**Add steps (Optional)**" -> Step type drop down menu select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ +**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ (make sure you have two '-' chars) **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: *aws s3 cp \ s3://\* ```python @@ -28,7 +28,7 @@ Then add the location of the file under the **Application location** field, i.e: **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ **Action on failure**: Leave this on *Continue* and click **Add** to save the step. -![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmit.png) +![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) Check the **Auto-terminate cluster after the last step is completed** option. Since we are looking to run a transient cluster just for running our Spark application, this will terminate the cluster once our submitted step (Spark Application) has completed. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 3bee6b8c..e1e4908b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -10,7 +10,7 @@ Let's discuss the right setup for each of our node types:\ #### **Master node**: Unless your cluster is very short-lived and the runs are cost-driven, avoid running your Master node on a Spot Instance. We suggest this because a Spot interruption on the Master node terminates the entire cluster. \ For the purpose of this workshop, we will run the Master node on a Spot Instance as we simulate a relatively short lived job running on a transient cluster. There will not be business impact if the job fails due to a Spot interruption and later re-started.\ -Click **Add / remove instance types to fleet** and select two relatively small and cheap instance types - i.e c4.large and m4.large and check Spot under target capacity. EMR will only provision one instance, but will select the best instance type for the Master node. +Click **Add / remove instance types to fleet** and select two relatively small and cheap instance types - i.e c4.large and m4.large and check Spot under target capacity. EMR will only provision one instance, but will select the best instance type for the Master node based on price and available capacity. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleets-master.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 23bc7993..043aec6e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -14,7 +14,7 @@ If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid goi EMR by default places limits on executor sizes in two different ways, this is in order to avoid having the executor consume too much memory and interfere with the operating system and other processes running on the instance. -1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the spark., +1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the default YARN configuration options. Let's have a look at a few examples of instances that have our approximate vCPU:Mem ratio:\ r4.xlarge: yarn.scheduler.maximum-allocation-mb 23424\ r4.2xlarge: yarn.scheduler.maximum-allocation-mb 54272\ @@ -26,4 +26,4 @@ So we can conclude that if we decrease our executor size to ~18GB, we'll be able ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) -Our conclusion is that in order to use R family instances with the flexibility of also using the smallest instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file +Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 7868fa9a..967c8dfd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -11,10 +11,14 @@ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aw We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ -For example: r4.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ +For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ However, at the time of writing, when looking at the EU (Ireland) region in the Spot Instance advisor, the r5.2xlarge instance type is showing an interruption rate of >20%.\ Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. At the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge +{{% notice note %}} +Spot Instance interruption rates are dynamic, the above just provides a real world example from a specific time and would probably be different when you are performing this workshop. +{{% /notice %}} + To keep our flexibility in place and be able to provide multiple instance types for our EMR cluster, we need to make sure that our executor size will be under the EMR YARN limitation that we saw in the previous step, **Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 18 GB of RAM, and also have low Spot interruption rates (maximum 10-15%). diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 24b87c64..da658e88 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -7,7 +7,7 @@ draft: true When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. 1. Open the AWS Management console and go to the EC2 Service in the region where you are running your EMR cluster. -2. On the left pane, click Spot Requests. -3. In the main pane, click Savings Summary +2. On the left pane, click **Spot Requests**. +3. In the main pane, click **Savings Summary**. ![savingssummary](/images/running-emr-spark-apps-on-spot/savingssummary.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index 5c2f70ab..3182f572 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -4,14 +4,14 @@ weight: 100 draft: true --- -So we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. +Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions, but when we get started with EMR jobs this could be useful. {{% /notice %}} -Let's set up an email notification for when Spot interruptions occur, so if something fails in our EMR jobs we'll be able to check if the failures correlate to a Spot interruption. +Let's set up an email notification for when Spot interruptions occur, so if there are any failures in our EMR applications, we'll be able to check if the failures correlate to a Spot interruption. #### Creating an SNS topic for the notifications 1. Create a new SNS topic and take note of the topic ARN that is returned diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index a9c0b36d..7e56363c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -1,10 +1,10 @@ --- -title: "Verifying the job's results" +title: "Verifying the app's results" weight: 140 draft: true --- -In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that the job finished successfully. +In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. 1. In the AWS Management Console, go to the [Athena service] (https://console.aws.amazon.com/athena/home) and verify that you are in the correct region where you ran your EMR cluster. 2. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index 15680794..0f79768d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -4,4 +4,18 @@ weight: 130 draft: true --- -In this section we will use AWS Cost explorer to look at the costs of our EMR runs. Need to think about this step because data is only available after 2 days. \ No newline at end of file +In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. +{{% notice note %}} +It will take 24-48 hours for your usage to appear in Cost Explorer, so you can plan to come back to this step later to check the costs of running the workshop. If your organization administrator has not granted you access to Billing information, then you will not be able to access Cost Explorer, but you can look at the examples provided below. +{{% /notice %}} + +In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=Name, Value=EMRTransientCluster1. These tags can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. + + +### Analyzing costs with AWS Cost Explorer +[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. Get started quickly by creating custom reports (including charts and tabular data) that analyze cost and usage data, both at a high level (e.g., total costs and usage across all accounts) and for highly-specific requests (e.g., m2.2xlarge costs within account Y that are tagged “project: secretProject”). Using AWS Cost Explorer, you can dive deeper into your cost and usage data to identify trends, pinpoint cost drivers, and detect anomalies. + +Let's use Cost Explorer to analyze the costs of running our EMR application.\ +1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ +2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**MyWorkshopEMRTransientCluster1**" +3. To diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmit.png b/static/images/running-emr-spark-apps-on-spot/sparksubmit.png deleted file mode 100644 index 5e31e0358f2f7d08bc459219a625c62e1a889bbf..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 14817 zcmeHucUV(hwegz+H3vRFZbLL%%b^s60Hnt!?_sX-OHyrNAwvdZf<~*fc^0V9%UT@HO5D!mz zn((FvANO1+=*;C19v(5w*8eta;Ja%)JVxQxr$7#o?hB*m)27>D+N8~c^nD}G(6^=a zhNJ9HYOnI$B})Me+zAgaw6_jwX&o_;=D!|si4 zIQqQU|J(CJXngVsktd+jypS&I<%7oQ8zv`jcUT;knKF{)K$aZBAe&4yhdFM>o-`AC z{%X1O+d`uu72wO$ILy65}3e`|wB*)#};mgkdM;FDQ{4PVmcENj!m|NiWX z87Jg#&rY|RvHHz^YLs8D4H#_*BM*F_da)*yI7>?8z32Y4Z_Qfdyw`tqDb86GPqN2@ z(HlQ8>1yD)H5xUInqEysG!y8>W1F+RXJQ}TTKdpwT4z8-Ad^KzJ~rbCJijAd>v7bd06>A+}=?_B3 z^Sce#K*HOOnAUQ|U*IE-y0SSTjW*n5jd5VOHXJ%)-9#$lzQqax9!;8-mshzXm8 zF+gOO(xK~?#-)adR8~L;YLKW6-(B`$7P`s!;QD7V#ekbfIxcz5PrKr&XK`gRj351uR65vPqcv>cQ?L#1Fbc+BwL+8mc0& zWKD?qbHM-|yzaqbh>MS{cP_f@b%X0xK!%5*&(lC#DqEE_aFo3k$f;p_+}gmr=)_ss zFBm`uwoL@0`|oC|fq^EmwLrFjzo1W?ElHdI!7c#Yjy@*9Re+Fd^0eioq!<1lqc$-0 zG;why@ajsih(DSc%w%8}WHLYM!vC?RNp(%@8{G)^G8-+<6dfP5Y^)@WXWrgr-qwv^ ztNFLf)X*WDd8S7dPxvXpy{9&>L>_V3TgrYzpbBgl6)*NZti4grNke$oeF$(ws>FH` zYpZ^@*q(0yl1(kJ>297~KwADdccIhQs&(5Pwvp)gveJeHM<$4ImxQ^AT=RYXm`&Cf zG*(gs<_^0jlrOr-hWD*tbr9>!4Z{kc-j;>nEgRRf&>LA8=uN~A{+uxuauWaN2DovZ zjAoM=N!W_bE)Ib|heNJnY5i$Rl;w#s_FUQAvyUnM-DFOsC;+{J$gM$dSuRn_|)GjJFJLrg|Md~ihns2_*y0Mr=o`4e0$Dd$fVpT!I5xZ7sE~f7-%CH+#NY?6*Go z#P~@g7=;KI&egv>*>vkRo~wnt3OrY9|FuIGX9O#Ty4YJ$bsHuny2y000qyx-bF=d9v*BX56Z0@H zn~X&jZ9^SCj%KVDCz2IqGq?AqxHD%IL15LsabM;Be29sx0Id z!G!db{_3)lF>#ePgZ%;XA_^|U`sh{c5|c8*3dN&9kE%*4u0Nrrml_D^A0U#Low!MF zuT7WRrn^H8vz!H)&!tnH-5MtPrTQw`!&)nZRbXaUGSpGqAMg|J!K#?x0HleP>!);{ zUA1(+bq9ZlwFdJY)g#>BYghYEXvaoeH1;$I1$9C zOT=Zxwmk`%Vpcy15v@&2Q5xg}B~=y)pK(0a_8@9RDtT=Xb^cs3v^sY$u0UDzfpf8m z74584ZsxH|QrMRvUltDdABoLANUqgh^@#OCV{Z4RYS)x35?cX{*vm@&U+IU|49seR zC{w6`Crbxa4#Jv&xGyn7S&f9{X1{?XddbB=>Cj*0yF=>^=v-apD-==j0-bk-$QRQd z$LTuCe+%JVswNv8z$zMBjQ|I1MZ{K?)%3BLQz4Xj#c-&S6nSRYB3LOvgGRLyao^Cv z3*=l!Z;ZP{*siyEUusT zQ%hdDe;xJ8I(hlyNj1>n)e9*HvwZkdo$b;a0Un26mF89GdZR9>WAjE!Dy^WW4QKKBrJ<@+-u}Ed{zb~7-51cq zW>?M%LxWn3=C+#+y03;f7$;V&U*D#jt1)XVy4~yWJ6u07sTH72T97$y;VI|Bp!Fc& zN~&!GxyH>XO_}q2IiI7_Cv6)D*G&Fs37QXS7!@^b#&4(Yw<1}>cI`GXt zMYt1WjICS;hfx);!6sI+;m@VMR3PW0HO@aq+m@=RPd%Go57G)-QWD)dYQMw4? zum66P=c8+!*#G*$@7L=zU?0~-Qr>m$4!ybq3*WuX`E<{nn3Fx*>h{HnjqHdM`!u>M zbnt#{5?Ku_1}+L`6~NxFQG!-Lk*wvQl_+MP=|K|Dh$tv2hZNtHu7>;Rvbrn2s}Yk- zKPo0I4Qph)84V5Ztj|To;z|+Qfb~VCOLL12n;SOnX2q28%8)fdlY?<$#*g`PicHfk zcpoG+)l=xju6t4@YKR64!5wQ;b>Ez#jzuy}C|UCeVV1V%zFQS7n~b0^Zx8Pz5V2A^ zsZbg$t_}3P#Fgp@uR{1MLBN%d>Tz8}gQRbM=$h86x`H5o=C2CFQgPd;b|1We!Oxdq z9A|%OQf7oUl5n9M_|HA>lrHh z3wt6uhH!AODJJZtpIPUoM2@-Ul(xmm8>r#0@n`W?jatIHRUDck!s?1@laAT(<^V!% z%uI`rsi7=Y56^I$C3)?+T~2`rs$~TFI+FB*m6{#6ESHs+L4g2!-&;UOysybSb?CwK zx9u_%=880uYmG&rt;cR{%q^VixzDWqAOaY5g7}~pG!`VX0qr)XDn%|Ew1Mn-X&nv9 zfmZ#1G|Pr7Y2ybGj1bNCgZOWcF6_joUwy=PZ%C)rZcqH#M#Z%A-B}OwBO9LiWi_`k zMqr1(w!Ng8>8CEcza8Hhd5cMH zc(blDsQJ1cD%u2}3?GY@m~oR?GFDQ>BiyDH1p=hn!*dw2HNbq-Un|C=}LVsYVR{fV;|UdV51%gWZhKPi+I@T{PokbhtP?U>c_s@6&1M> zli`12H#p(s?A0S%)e0(vbmPB_31#-M_9uE8UmEG(Kf<3{U#JGt(H{8bnHyksaqwHA z$`C^cdNwUbM8@9CJJoIST92$dW`2WLVat){L>li2J)%<=`uTvU*?DK5xU-EgP>;~p zkLdx%mYn24w%2f#%nMMy=-%mQiQ^Aot`FZEQwL?REvJ_v zLg#=Mi-xxzj>0={N(EG~z1wz6{+7-T`bl=OOgw0eQ7HYah6iFQ4Do*G7buJvE!q#f zLW-YpV=c9gMOaslzopdfG%4%1(={$wlkYCAm~1@p1}Z1IZ$ZbAY~DShr%KnJhX7lB zQ2c&rIdWFYq)$f$0YT#OOv~>h*zG391;=f+`C0vE1OKI1=h(CD*R_8)`Cp2H=zH9F zX?V2C=zq_(^n=YzW67>#TmGeMq-7Jp9ar=& z8xvCpU!GdL_T~!OC&DN_qU%?-hJxu5Gc zTXWf;y0LM-5prWerjabm|6w!pG{Y9V{<(8};Y`m1uhGi^kej;e8B;^=YOq+17 zC5`J;_CpwbFN%z`VP9bv(E64=QLCzPwuhvs&OIxhgqos7gXwTahnle5S=_09dEl)t~gT*SE+;$+Mu0T*bVVxdiz#qfuN89{#Cr-J>3Vfwf9(mL7GU zTL`{IatEX3$Qaw2K)ITh4MHQp&Cx?+ROzbl7<;XjS^wUDyqfD2UmGZdF6Z@9UPvx( zYGeaRw6|eRkPFc%q)s6mwO^`iXOMH>O*@5nbF2aZq-&P&f;@xIo+Zfoy!1fwHBh11 z4C6c=A@ycwtq5ScUXCaXU>T&ok|=n@4;akb_pT zp{mHlYx()&W|a_EzFJX0NAyrbo?gh66_~uNug9`E&ma_Lx*G_VGWL z2|bOZe4$PFR@@o`oNNsP2}GXc3V<4^0)A!*lM(Eqb$^S&6dsX~;>}S;7d*cG2;yM9I+G0;hV$Z4&b* z+)nT4>*D$e-m9{pq!abUc@pS`hEXSw_`|#%K?d9?X-X{4cJs(h;+D_^eg$l~6+r+` zgay}`{NkGW2ZSs)W2{=KIfm}q9m;cibkF~j!_^&1u+_(0H#O0AD^ASs(j#X)o)JNtJn^5Jz zF@*qNoykZnp_K5<0GFiJ&8!SSn1P|Mwm(0PDB-#h^9unxuMsG~`S}B+rxUEE$vg>j z+q&lmNh40GN3N=`U;%NqiyNKe$0fgu2Ndm5SpBTV|7gpP(Y`qubfhC+J;Ju&s}I_Q z66=6&wJ-=O+KD%Z-Q!TN7h7moYL z$f=bbBBT#ie7bclTY?^~X}AUz`w-dvgw9qS(pCzQKj&8~9`x5dK4@RFw~X3#v|)h% zUd0Wpl<>;c7<ECPhiMCE=|b7sC9{-eTY@s9MkXijO~46AJ=T;&xRVeb0N3S%*?4=UUj zAtJg^-x$-&UkaS6hK1rTuF2vijS7*mp*T|+Z+YBOa5&leSPE+ZE7EpZohj3#J#F}v zS(}o#Qr8@_u^O~QNX9e*K0XBzY2!y?v_{Bva=ueE!^G=y*NzQ(^z1D4spcf3H4sLm*J0TiUi+v=wddRKy_q~bt%?R2G5 zC1Joc`@=ju^vZT~vru74#3PN_JS%FW7Oi^$1y9f5QTC#zaABO4a*RwGT z5-l6WEcYLD6(e|%_AoHLJPC(H#rBRQ-Q?gF;H>5h*}xb=q)oIN3>7+7gZylodqOUB zcej1 znF>z6B7{ButQXh#nY}W^0EQn8b!AXlvbzUrC#4EQft>W!LS-{YM{)R7Qq7KtpHM!QO}t69YxMH z#tkOr_ZA8)txLq)x@1%a%hU9Qc}ESQO6A>0k9}cT#n~9wG>QXQMvPrg#=7m>jKa|s z#UzZc0nso1aQw@ID1sk=#4R{u`f@SaJAtU+V z{dUO*ffww|$1^fE1?Kkmr-hZ+g-srHKbKbQ8qE?w;eWj;p4|1ewMg@t+WWHF zc{HpWPp_-52}hPMU86^whWXt>42Ls_n*7E4dB3mWQxJ2gRjS2nx3tLy8gc$E-QTa| z^J*lGG1Bg??8%2QU&gItcb*& zA4&uj`jrPEwMkW4Li=-$-j)2&XokhUg!qPZe=Wz-D#CE#x+I&2lcl+&6r0ima~<`= z+8TR2_T;MH4&l3DcbGo`>ai`;gmQ0cbF?lQSHPH0_Ljj%#oMA;Rdk=LnxyXpGFuK% z$e(k$Fa>Qy?_XvKPhMF!V!n#&%AQ?H9eF)(VhCByslHApRk=V$)7W~?uKSFhDU#}y>kYn-Pf;L(1-I|f*B zrs&-L`WV<0xaaa1rbRxjQy3Sklkrrtv~+QAtENKWjU~m^{Og(|-V>8rT{=ZaQ)-(E z)-p<-jUO7iB-Om^vJ%tu;|H+>W@==$Qg2OIX+>795}dGi$w$k)I*9X6>Hcl2k;ySn z`CdSg3-6h(>Sp3r+aKpursjq02YLGjDh2n9yB8<_Sbnv$_V*5J#J#BpN@Bj-uxE_m zL|zw}RV$%3r)P*CO^AR-Z?=r@w_{&M=DP_58p=SoZZiqrl9`y_)zDWv zIi}C)rD9H%oL-$@5jI z&(Qzl5aXY}&9MvH&b>7IJ==fm)&DGR`L9m@iH7`_O#kOqD##VG@zrVW?N&hiTd@KT zBxZm5V`*GD7M``Rz4})KZU0n7?buf6``a{iyxO@aMi)2a--@n{?Eil6Y5gOKjK9?S zd->nbeh0>4e#$luTtMO6zHdSRo=*Y)+Tr6J&JtQI=>_rzw?Nn>HS*m&%Fj)qH72#aeZ$yWi zSk1S^bx~Oasselg2lwxRvv3I3;%g<=yHQ_L?(-RW7PAvAKfX;~RWsO@7 z$1miPH~ft!7FN`f)WGJATiF9D@=~BUE{ucw35Jr%%`st7b+NS4E5-F14-Z^e!s}~D%@^b} z-q5G$OO7~xM%IS)2&t%LQ%hTJOs`~Nyz0^%Ga6#U?5~W?5*yC^f15jryuboMR3?*10V&9FR+01)-g&8hrKEzfGfOK=@ z`zUS@_EBjwm=jv6^?n^&WmguIm|Hx@~~}<$lsr zmfjswp-bry^gEy7NCBoQnzK zCQWm%8x!v{-KuARx_B~suIJ|%8XB;2O3j`ZdFS9DZ+@gHY^dZXh&bfvxV}6|hRxx+ zWkb39*m&$h{l4+R85$Hse6vy#rW9|x#P+xD@$il7f=LQPT?GXaDhH&oGCLd8#9M=5 zpatO&my?BYwh^)PYxyH)(iZ1rFEWCqu@T#pXQ_jFDGKT8T^ClRr*M1eO8m%8cahJk7WnBhJgrqggHnE~>_x^9os^j3Zkwwo|;8q9sU0ILwk|k56i{ z%CRVhZeMq6rF6#{D$Qk-I#^TCU8oKDfx8q4Eig)LfN;NX8%ER}A8zD0#`rK+#L*ilpD8XrAtN~q3Kk{kx9Y@b zN%a)8W8n?Tl2r-l*~W#}!lE0|vTg;TpY#2>kBzTg-IQa%mKQ7v-7IcVPg55%1_SY2 zJ=%x_R8lPt;ZbfU~56qp!PUd~6Z=AV7Q3D$O{b6*<=wLrQ^XdRXYv68DWUOLbCTg;|q&sUQtu)e! zH3hxNoE=yR(Pd5H9CJ_X>sB_IhNcmP=*L!n%4dx=ZkpamB?Ax4#^B(#4a}DT#R%KB z;nK9R#+Lc=5NLuAHqLfE#$ddR$uc_Q?L^_#jGXe9VP2+f1eQ9P#Et zUTe2l&ONf)Gel9q(mEIV9)Xk901}g|S8=v3*)uW7G&gPkkW5taMN{y7M7nk>bohqSU7 zZ38cK4qXfM^V zzy#3{$O>T--r{bZ?Rg2%-!x%!mo)^W#*PSSLBBhT>3+X3Xl5xL+E&FEP$7ysefB}5 zNsU%Z`7g2LTP6jXA2%4QOZo3!V4uV2SbA%Np=x5{{3*$);YCWv(kRSAxN#i@5 zn0oN!)7BFyg%pkXbg7?JLEv}Hn-Bc2m{{n{Gkn{6r57vEkB`}PFl1c^5~1gkBe?JG z+N9kg@#^s}JEeU!xDR9QYw5WMSoM!neWsevQ5+4yYtpm!E zUnUIwmIh1n#9fB4<+N9Ox8?hJ_4v4uD(BI2Z<`4f=ab&L^5+C_Dn#Qu2G=@b4EHFK z^`%l9=%qKVC}Ehlyg_Ogq#wOEACN+q=j~1RzsT3P2 zijehEs#zgICkja7?evX5Y4`AT%oB;V_X7|I7!4~Hf6Ao^8*dxZS!s#7sASv=4xd-P z`vzyQXW|KqoR{t%Ql#jM2Op>0;bvNB+-&uCLi_=&KF(;3yUz9?qB#4P_}nG_4Ey2t z96m3Qx3W|dN|0WF-0sKd{+6zGV6^|?dbB2KRHhGVfrPT0V!S-m&}%f1+F?xB;^6U~ zP;kTpD2Ch_nI2p~_u!^EUW3+z!d{&*P>-x+1_g{==(Zj&qId4}ap29e%XkncvD4n^ z3>ltou6NOdZ2p{LzfV{-p`pyMt)@;5CN4a620>fzThdL?CeA=MQ8(rZKVmUOvm#cRg@wU6z(+-hGPDpvCRxe7D9OAl;#ED^_fr#!!$*vc5 zFTao7lP;_>Zu>T$SYvB6KumCz69$|^RWKfotv`It`C4?4RI;BGG-4qL7($Ya;$Z>` zNa+AyO_HBw&mH4%bbQyM?}09lG7y%S;Kfa+%99pP3H&-L1o#Mz6I&eI`s7R^6vQ~w zXjTc^5~q7UCPnM}9gm}%`?-+qokd|qwl0_Zw7wPQ@B|_aR7efOg+iwmfXFdX{|?utwKu*1i%WO6QNK3fl8TdmgQUd~UTZM?8PmjMjO7h# z^9{JtMvLGU82J4xru|rk?()?O)d0h%<9Ry|Drt%UyowpK+Z{$3zjS2I-6%&t)@i+z z=AlcH1Xoq~7dHMpMeA_1zv}7g@H8uqy*hpB{>}tV^{Zh^;ACIp#Pm2jBLbHQ0k;T} zF{2zErRSYsqWra*Ag@8hCH`scxGk&!lOiMXpzv@j4sfOJj60R>-hHOM4%Xw-jWkiY znL2B6rqW#=(t+H6J}p1yh;Ch5WGbF7t8e-KO`qT#GYU7*Ey+lnh^EaKX}FbPx_<)I z_EvOwMwX0uri2MLFx6z~#w0(90$Wwmi16=hzMtRHj6b1?B);R@y6-o1LnaV0V8w8} z+?%O<;y8Ovz8_Sm6oGr999H!C)HAr=u3?AqoOQX^M+ZXqbR^$(cq%^dA$SV{osbS? zIIQlsnd&vS%jLcg+@zmgkLz+ncN#wNeOFd}MM(i+7>GuY3K4R;!KMfBm3b0lN!K>H z*sEy(mU*1F;(82w>TSLDXt_0VJR1x_+13=YRNG1$VOk1Ka>p{e%||Z9Ub+tS&PRuC zPDlGsHFtJPyGEO~!ell!ZyEo7m2}U0GQP88$wsa{_d(O#Vog7Gaos-7_UTv}o@JVI z>($Mj!Hvf~i{4qDFMoOXd076-;Q@J9sVc+V#V^xgjw3DIEI9PeL3Q_hnQ%)jr4@h1 zhcb1kpFyc>M7DbZD{EzduBJXbRQ_y+O@oS%momvPOu5g;2TxYSsz0kMKqFH3Z#wqr z3=nN7=L{0Re_;*~#v=WsfGyoVX06c(YGPl^o~PvMRkxQj(N@f(*e^GKPkVC4+~P)u z{lj{@eyX%Z+cDCYM>>WxH6XPBWbDLo6Sh7`?ZKoChM+TjYOnCvQe2ND?!Bkp%?d+4 z%Nssnd^+l@9w|PCGR%7Y5Yz@7|7(})fR%`zBM2soq7+f=ao97;?dDb?1W<247#%o- zmd5JtF`V~Okv&9yO#ixtp&+ElLDJDr#-1fa$2 zZaW^1xNN?K1bH|}1NiyO5);NDJhohb6EgM{oidX1+Ao=Vl3Qyy;L4jrsT&Vl#b;XB z>KI)}Fuy2S*r)6)-zPCN)+h={r{bqrIYtGNjKlvnw@T_lc}OkQC%oJF&&*D3 zyoe2%PS04%v$xLOnE-=+oTY!`SoYJh^c_kB0U+6MhCgOf0Hn6hUiJ|~KO8di9qe$x zvZ7xyxgzgJLP*K&6v-Y!4eu`9f$n;GJ$c*>?PGVty&D~Hlf6G=P_ZTf zIT_jF+!vhkLHt2@F%aAs80Dsp7TAbfp%pCBnKR^BqxjG3GlbfgdG>_Ls12fK@B6#D;l+G zBreDiWT47P1uyGP?;g7Lsz|u|2a=NY%0N)P_JTCp9;x_9?rz(Uc_x2cmrI;jgc_51 zG=G#rt)92mzu<`YY8t!ot+U%BDvQ>5t3`&rYE}&vR;_VGLJ%8BPOV=vUd)Q?Ssl>+ zpRVk)agb@b})Kri>JJo*`IfFEd&1T>W;s*jek?j_GZ4_BK<% zv%#@2TK^7-(gTEKi8c=9r&lv|x_G9v8AgzgVHLQu%sc-B&+3*mLBYK#A~=$9tG96H zFZVs6ikT&K5XAssAjHhk1SJd*cam_B8i^l>7X%=i4NX3@oR1G~XU zp{w{T=(WKco}17BYp1+rVuxcSnCqJ}L6^%QN~QRf7cjkrH88dSz8hlJXy|65aiDW! zyR~8{4+X6an?2VdK`BVU6>x{OcDhmAlEoeg2#{1!tJ%eCD*}k>rMQal#2?#@|L){N zTnO1u5-Bb7(MieHB87es@_dcc57BN*;q2FkT)qP9EA8?@Zt9{}w^*j-5sk6Q-d3z1?vOhhr@sry)# zewfqJE~?Xa4l{3R@aM^#9j5t>m6w&K${Q$!K2Wn&g9v=`=_>?;{N^%!mXTVk2ok_} zUNPJg+QyjgHmwGunnR^wYj~?TTj?{y;gr1eyP2QspFD?9n;b!f@Z;(1oScK}A`BTJ z%XE-8qi!hAUMY*C@aiMS2z7IZ;9p(mXQ;pPk#XJ^IYi+-y29XBYVda#l6Us}3=DlWF6u;GY|cEf zk?fq5IN^%~49W{>4~_L2A$mHxdKC?Z!5@tD8~JZPEF#zVn6%6Ih@ zL*tqviIy?eJwjc0;xW<`PQksKX($!S08Wzq5nR#QCFY>I`?&-~eNa9}?#7gl^}c(V z3)Ux%_Bb;p&5>NxG}h)PT)TMjCl`p4sVw_&Tl-$#4*I8Z$SI!ZyF-5|?&9igBKOMe zKWwWyfq#%A7iSBR1L%}3CT5gt2877vT>Y!HlYiQQf+zs7o`iU}^;@#cHRUX>*#EDO m`LC6O{*TVjmWxW5z1vo8H+GrL=Kd^`$J)~NRH^yZxc>$oL?jUa diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png new file mode 100644 index 0000000000000000000000000000000000000000..d5ebf3e1e7c7ba540bbd3e3368d6e9119baaf6af GIT binary patch literal 15091 zcmeIZcT|&I*EbkN1VNhgX2b$0(nJ&tMM0!@0YxB2P(-9ds7X|s6e(gs5F%X^NdW03 zbi@KuB%y>Zp@f7UNJuil`+n}{d1uy|HS7DnH8X4G{UaymI=Rj{*R{{yzq9u~zr08p09 zy$fMyT_5r_x)}ff@V4!J_I3HZcLM+}JvF|1<$8$AGX7iAEDlCk4S62QhT7flGJBt3 zzQ6J(_j#honTz9xRU~s#63R>(-;aqMyyt#*>#nU)i)4$McXeG>W%0FEwMT=ZJjaU1 zddh=dleeC<1u&74-IIsPkH4AV^KEuZ(b96en$1Ls+2QM%yS+@}{qqvfuI2SrjX;^J zKxPi&cK7pBrmxQ4EFYWpr;wQpGL>WR%8k|hO;ITbm(%yB$LNDm`Y48e%P{ZHb=9vb z8Wur!ptJqy%ewow$m@E<&TB-UE_$!;UbDAfEscO+xYHv&ODcLUE&VBOIZn_Abm?jE z$A3-h*6p8QDh>UQ$J$P2;>^8t!^o2O?(R5ma)iye+yi7o9);c+u6xFZoVgTWgXq}G zBSltY`d}+elhMCAYj0H1mVFQ(5n^EYa4eT#U?6P^oD7s}weZ3yJL7h08ekzic_f&2 zp-&*0dH*ZpU#}TkHAbJii0RW!RKoZ?`PMxUTo)WT8CpbJdWgYPjS{z#S-sB@DZj(< z7sqsbFcYw~QXbHRWzXwuEO6b0QZUXa;JK7jyIuphn{i056l0m4ft?6-NX!E+$t{Hb zB}~&-nKu^EY6UwfKVy&V1mFnYNR1~WYPOyQn{)2`@X--~$b0*NtGaKW938NvF7NCN zhBkekI_)j{{k;wzS5c_zDFtTIR0X)B{TsnS%w$U=p-PkLNufKqfpq&WSSzkwCLQvtO*M}xws&*qBe>R64fXv!Wp2aFKsHIE9#0#QsO?O+( zIXPu_-(z*Eugoof?5&s?^HjRe8oAzCN)F7u8~}`QN}y6kCrIOTy-)Ri!Wq)_;*p~h zpQluR{#ma2UN6cwFr!@3b0XMMpKE#fTknPr(s=~!iVdQDEb8+tMHV51snD)&?AjKc zZv8>jOsHT}=7&j>-;0G*qWi1q@b@&nso90t6Ei=*_0}m!dY9@*IwD43 z*B1yUM8}#oQ(^QIpo4-`uyv5rzSF5{PVxK%agmCXnveWM185Dcqd9Uh%h~ zkm)wDSz+A!u0=-&)^qdvAP5{=>f-mP{2S6V^;Z+))xfWE29`7!(ZML15%g_;?*J)p zWJ@Qd7~9``qLTq=!1I1b}Ztj zuGWj+vCOuZET*;>;%`&`O?M$N7-Xgk^WCdbuTZ+O7j9>UsqTgNd)?jXa1_}E1ON12 z;37CUxadC{Y+|MVn(FhXD*OZ;KN0j7*4)44RgY?-nlPLHh8nlpj^897=9bSU-`_T9 z`s$X`B3l37;Id$e7-Z=mfdsMN@YM}_~)^^=3 z|NKRlv#=y%7jUq-3;|4OPr%l${eiCh^!in* zZ1u+`r8>`GAk~wqYm8IHTJv$P25p%)0NY$ET4oBwkQzQx(Ef19kSU=A>=ThlQn(a* zgnVMcmVPsI2t!~vE+Q2G6d;6y;|P~&@%xZ_HCk4BHiuUxNE?(5gt&`vB>HZ6@y3skVU=Q$s+hZa&Z@*`}js2IB4Jo>U^`mo-&nl*{5qw9vKZ#tHjid5(xI9x{ zzQ`vIjVLwf&{e|qX(v(Nn@)&gCsuHSwH>fu?IBnBQ&CF+cH`e}&xZ|A`b!9ulA>xs z=%d)D=*Z}r6oGgPiL3p&XKu--(8F@%e{s2m@pd#qSId2gIL)ywn@q8zF&&oPoovyX z!GU~q-1-x*QfJ}qh+AhGfJzY^T_r&1cX%7t6t_<$2_4yn1{)i&%ZC0CwgszHpvsQu z+>d%P$L;U(FX)Hd$nVVoWyqBHIcanqXdP-^J7hLG<6EZQYIZZ$7R~iux*#%sPL`SlzcC@l{4Lj{q?HFn- zjkowM<`DHf<@l{n^1lf`Q87hk^Vh!1v$oZFwA-D zjVQ6^{iv!?WbR=li1Q|JBK9keSo?d^VCuaHw0u2&M%Ec6W;rF5GZB)7X45*$(utJQ zVlw>s9s1su@ormTz3iz?^5#25glh{@RW-v+xp{KgCwR0167Gyx1~zKj&MUX8KfN=a zkmP{PWxIU(s@P5IYV0j3OucB8b3GqX&*r^lD zK_3CA>E2DR5Imh(7?9J=vods7tRey)}G5iK6g1^`@m_CHsaOwuV6MJz}h(epYC z;keQA0(=e7%-{qyrye5$n$P`51(z~Pnx4%EEsS=U(w;-H&bs!jO&*I{HZ^?8dWdhw zh`06iaGIL6^VqjEF(|3-n_<0oHTUvvG(OOG&Z)>D7)s{}@f!!vga)TEB92|Oq=s+L zrfBc4vE&5d_{E{rt?wOKdbrR~fvo}as1w!8!%N0o z?=>a@2?^}kJTX}gukj0{&-D>^?GJ&P+pghSVopN3Vlz(?8=6Qxx%7ttQ}Wz;RoXhH zN!(MXc`sFIH<5a^oVV|u<;bl6I1tUXl^UI;Z^RO3>@##Ej~|4PTOah;fFmgtRpzwq zH`N|1O$)phbz(jEu@k}@pY0Hp$-?hDwDWD2&?BYk;Nq0qO>^;W@yl)A2~VQ9M3Q^z z0~(##pPVGlPg*15^O4WD;Rf%-q|!iL1-$-Yv^-M;<{*+d59cO)bjdrE;LqIXPU21S zMz(77bf`D=@pDcdV+kW=lpji6`@O$=czx6a=SLfT6(h}5PbXrV?jUF9Rcym;2CFwd zmdf-gyI!BKUy#?bA*7;`ji0nmray-XYbKr6jFq} zt-Kdb@|_|rog6U3`Jr}-l=>W8?!X@4{6;&lHv$sr=Q4h}a{J|Q8d_co>?0-Ac^AP_ z#Q8wO@KYCleu2|9IlA1~UJ`GS!JOjL?y!d43}cD9PPelj^YNY5n$`9-vH%JnghTRq z>=C=MG3-qbT=xhOEjTn|zXQ?}X1ul*9TZMZq(moIb>b{mB}Au{7kwy7`r#KZ8RCXh$m*2-!DE6q%x;v8x<^T$;?eRS z6=JC#gZC)dhRN|SqYr+0o5^1_V9*G6bsk>Ry_#!Ug z3G6&Y?W1*b92F=Q`+S3K5`LTrFg(t<8#s0B^NLWOd&0_&;N~xTNWCJXZZo>Ug(w_Q z>Km~}0iTq-9f*O=*QU1I_chy+vh2y5p^ML^$3Jd6I&cvf5mRbCrdx@ip6`k{bwByj zGkqB%AoDVAUs5~Ee#v49E0gK{m)QYJtY*U}e;Mb0RPLZyP3ehfHmO6cynwTjdzH;^ zMtgC>-4QWX699Mqf3!qDvDBr1rcFlj?Ro6b$-+6TYLxVC_C?Vn0N)VXIR}=Y_y05N zUzNyluV6p_{w$%94%7N?kH@ri2k6<_m1j$@8DX!L^F z0QaM$Hx?m=mHGz-TP>V7U$6eU8$81CcHpCLFR9dG#y8+8IhR~C>Gh=H2L!D8kl%$?~aF&6idSi9n|cm_3C;FnfAgu(Hd& zJ5=x8X&2iGy_8}w_A~9A`OfRU2v`uKwRGqyIxXe|U1+|q0GKdNlI3L9?Pi=kkbNI4 zUNg9JG%J_@n|w}VYnEuDG?JrhyDw}y@<2Em!*FqJF?dU%)$OIX{9F-MzEgQeT%3$n zAcYg(v{UgXQvOhzpgtQR_ zfj+I)zZK4QunRNoZoqj((bxG;#}@T#&P*POe$z)g_u~PiNW>;L;Ek`~vRveI4IrT5 z&_BbvXcMnS!{5En^tE1emnXK5HpmmBT$Nl#7O#o%u;-f{fluuxId5Ra*F5%UQLYV!6tJaREog4_RF1CGKxY^J&Bn= zaY)cl#h54wE%zkabxM_j8hgrEvR{6!%A;(tLDhj6{v36>FCJcgLI4Vo+P8VtomiZt zhr6OIM7BwrYGseEsYSh9q}S;5qitbljL+l6I*PQswRgUr#XH2xS6krr`6ly2e@ZIP**N0Xt$g1j z!j8^bI-I%3!l0%?J)^izu~O8y;m$aZIOFaZLHA7h?zayL0`~{)g&k+1qUluaK$S&s zuw~#htV8n<2tX2GxuNoWBC$4yjV0B7zZL+Xby+RIK!8 z+`|)D1weo*;xp@s6mFOAR}oz|1pSxhkZOEKw{h_Fr~&iqnm6*zx^u&4<+DA`1Q`@+ zkid$mwoB^aUJkF<)^b6iB0Y5s`rDXpu4bq2z-3NQ#+mPhMx9k^Z48PIiBhV!t)=%B(51s}yS(+;8aV`F3C-R`Xhye@hE`kIkUO3&Hcy9HKcfW^Hm|f& zz>tIp)yo@Ohd}T#B@J8vEgYinz!_G>hfwrDdgHSy7zf(H!51tpmah6yg9Lb;G1T;j zZyP@+6Jf?(spFVXPSATb$lPx9AuSJFZD?-Q@XPEvEF5Sz8C-<_tx!RICjeD>gg)^i zd%Q+(scNV7qfn%we1Bbmf%+k}n_r$=21UqR`0-Y50psIumeyu*o)=p*EXQ@&MC8mg zeSNbGoksF8pj4dinRUF@Y5`Y%k)Pa3d)|Z{CL9mtE7F*Ed%0^oE~@9e?FKn-a9zaG zPGiXu&^izuQom>@a=k2D#}2J{1_yO2?QZqrs*UhZ)vA=Yp&^4`pSiuj7+YPtF$0SWkI8|jyv$vUTNWQ=)Q$J9cl>zHRbUv^0*>Ch(q(}DJF#@y0ytrU#hu5BDyJgzkU(NWijA5z3p z^3y^;Hq@C#;b0;CV`p>>e7h_UgUU4wXT*eK_v>0J^nZ{ z;f3ETlEZl@kP}}M<^c?kl9mfc$$gPk0JYSKh-%#W?rfS6Jyw``_B?$%5HJr;ZjC(~ zlUobB+S1#d`yqd6>uul8;mxoJuX|FZXg78Rn|i*f-KmBFX33X)=0*Zr9(NFmH?0_g4E6%icj2Og6w{ zHfs85j?c$w#K!kgFUoo+eQU&P_!N#4Szva*x(QwQLs$!Vy!=&E;%6QqyHmU(Gcq}0 z*=NV{75P9O0Ub17NY@2{Yg)jo&*#X!+Zr>T>Xft6^|vfDF;#tNTgfiTu<|_FIlcEO z8?)oApgiOkceihr$PJ1CpZbhQc|%7I;g?7Z2F=8^IaVe! z9-bPZRK6o1QO>O?MxB@67mhkGtRG-=1-j;S85_-aA;?{{f0v*uv| zf|YRR7=GIqdaEkYwnrsnW$Q7XH&f4{kS_QQ`bmh~!YvV*(1#MwueIr0%NPDe4^-r+ zyv2%Cr=rvFdHI}4$rl?5wzK4f2UCs37-f?P#QJi=dpsCqADSF!Brw%l5-T#%yWtWw zlN+#w4H*CjQ(dmY5-7Ll)r@eVVw@mB0q79a6xX8d77yF(mQ-Qe+unMvMrBnBKz*6> za-4>Y7p!3DUz^^?XX{|ac$D)aHVEUsA#R_2vm5(urMYD2o*Qmv51702f7*F`u z8K~}<9-R^(3kF z-^+ge+`RlB@A;24n57<+j^6W8Id1G@`OK;wKc~nGuNP-0V^B{kR0KM0szg0tkCR#Y57RFfytn;5*SIXsdY=Gz*vfrb4A!hYY;GoL>Xo{82mjbweEwEzmu(H@OaqLr;=j+^MqnP zhQ}+Jxr6rOXzDvi7wHuRAK9~S=9#F7D5#GDH zV!5Hd422!j8V({T6HPFbTJ39%rBu#QsmkOL=w2R0*{&AGz1(*fferuT^DAw8hEA8} zjw6ADg!KmH&5SX_`Xwdca^1j3x-OPFzO8DW-=)HU6H)$_Thib-xKCk2??5}U(tdqH zcg&I&la*8WBqi+JXTH`R08l8~d0sDgZTr%jpZeCAD%zqfV;0yNb0!5EeqDaxwiOo_ zNqsa|j=S_{?A()QplYA6-2w(pZCmcfhmPooT?w0lfc%;7kycvc`OMd%cQUn_BIuCs z)6x}>J6N26E|s~FaPgj($*}MF=rp10&M&xSj$5(o3#r+S5gCiP0r)NZ^IpCTXlL#=WT=M+REyXOV5*miqCY)T3sHL6thY>eopD6W?Jfqd|yEADM8?v%fz7KFUw7i^hM9Dp?=XubMh5WsES1aQ0?u?Y2`!O~y0f);qZ^YZ)u8&}FZ2VJxy=+t}q>JPe-L1g!F5>YVI)vl`cIU+ZoS-nx|A>lo;C zY3nQOXV(`>OGLIgl0-0tIa4*5H_u@oP}3;Q{@4)V@#>Jb=iX(k=g)LpDvZg}rRM4O zGfg~8R@H|dyLDVpK!5#oI)%kAc$%R{rmjMa$~q=Cyl5*eGA@>FrqB5Js#u1!11*T) zfgnS|y}znI4R>g$K2-`Hm5t2`(Hftm{Yk{I6RE>%lpoIZ(zgOj(k3{R>kroY_&3HkOhi+D4vbFI)pLK zCaiFNOlRp$O3>n1rLWZN_0alH@JsYl@82dt-^s}TUt}; zJGtOM?No|jxtCu~+O_ytc@1m%`Qk&mLbi7DW?-t5$LdP7<+82v2ZK6g+ZPe40@3G{ zGs-b~sRLsgCNm4Yb~Rfy+wtr9*pUI(MX*8)AiM;D7=zx7c=>)}SI5BBBzd`mY&muG zn`^meN>_UL4SM2xPx%8NZJkO*k5c43%#>6OD$?!o`RJ7J+&NMzcuHb}QMn|_HM&c% z5}?2LJd?!pQH%?;NC+MemE%&e9c8hDHc8bO`lmmuog#Wof%{-}lTpzR>7|6Qp?;mt zp|n>~QJ)JSUFUzfk02%z8pl8XVjIYYbdix&T5n%eIglYTmly51j2Mlk8RQcf<%DSI z7NM*f{)3y-IZ50dGCb_}?!1mxpnm&$^RX3S0Xu*>pydl!Rxnuw+b6}}E7M(fygnaR zs8Z~;HjZ0rSVS3+kkN_hU(gmVwOk6l;KA@kt6^RR?=DvWFx?V zftXx?P<*Tszdz9HNV0IX&cBoH3Y=-hHQLd4>S_i5MwCtY-lX`(^i9eE0N#rdIP*SrU$dW^CdwbGw zh&jevyNNFsNgV&Wid!<*H6&3hT-iaS5U{2WNVd{)4fG`FWor_R@4%}k-(M6`)8o+t zvsGeJ`FHv{B2OUtVCbKU+3oc9w_a%m2EtO-A$Dc!(S23~F{Ovbt47@K?%vK1ii}O@)^3*EKM~~1Z9i>$ISB0rTU0*3}N0%6< zoQ;nsq`iRmCC4jqM)N<0dRXb{9Ykv1mHBX${c9Nv{kdOCgRt~*f=Qo1ZlF*$QCYH4 z(q;ifh@q~y5hE&zO@^3wLxcE5?n;R3_=c1b%a8MIfONqOE~~m6X8CUZkgkY=PTMS3 zmUnmbkt?E{4PeST+v5X3IGXiY)S;t*%>)aUhssym$cj$^0JO*dUFB_d1mmJ}@a7X1 z?{CZJ0ps#shFnG6n%;tp_(i8McMyAtb8_Jv-+s$KKW)mXsYCWPOFdv%t#i(?LT+x> z=l+#Hk+2BsRY*m6l7`92y#xtI9w{3cBqt|Vx0gWo9{!dvnyW8e(% z;Z@{n8A$`%Q@4Z-Ydh(4me^27v);&<5IUM81X)Z^3EoW-F7S!9d) zZw}DS`A-{1`QSbFxn1aiDL2Y||92qAW!x}tEAPSX%g)k94ssgxN`5wf?#9 zvdCFbjB)fiX_SU{4OsZ-*RBP6d2T^`a5zq*37s|(yGW~g-HEo1;Pa0}R#e(`X^|Uk z$sMYHvL3e|adUJJyqHxu3RG!dVvu))o^&!GUD;r*)91RPj@=nbn+abX$ft_24&{gv zc+JAt%xwwEsJKUP@-7HHVC3;!!7@0R1S@ zxLX3^UL1Z-5W*oGB|UgIdLlZyW`&SbusgDgDqAEZQ&Yg1r+!4eY29AdT!hFaj$RZf zER>It-oZInI<9^a(s4$|%KE4;IB?fxY|rS}ZEGU>Fr~?=rt2Bn4%aW632Z_` zhHvP|NB7GJAKXTC1xa{ylPWi+#vYn_|DEPP7re+6h z)%9n;uRyYKl5^b9rr1}cz(v{M-U0r1E@||_C4HHp_YplrLQAsoRvui0e5w9R0-u<0$_w>mic)@-|& zpSRH3;GIO$;7)wBvhsnxP*}P=@s^^BE4>hMMMP8bD0MxXLL1kzrfs4Y)`h5>GiveC zKmGSzQT|BLj(Y5?Jn-aA67w?4ghsQ1-+ytZUvtvECq9wfs%>UB;gCnwl+^HEsW@G% z^L!#9hc;=~n>9Xjq%t|Iw+$YZwS0{D?if7*Q??Ojrb*mrACRd;abJXBw!xQJFxZkS zLzMTJJT$sCQ-KzIcACdH=OxMX$8B66S9HIcps(Yxz*GhEaccqdUtCT*K|x8MYu;lX zz7@tkcr3njQR(tr#u!Yc>%;oOGF{VxhQwDEmwoYjF~nMe?nfy{U&oWLII8@vlR#;? zHz@TMx6hbTf3+x<7`lFzX$EEzCDmM2kXgF{`Ls`P`!M52iHBi5ACMg3zz@jH7ZL%u z;1wN>NMwh{71OM)Wy3!h_i756a+o76_81@N-t;eof2E>=9{g5q%A4F&AKR{b?9Jk5 z4}^j}9z)q>6~wlJhQI5uq&(F!CaMMxPu>kNFXNptCPDn2^(it4h_j zpAuX<8r8@Sp|_w;)TYIskm}8u+67vCsiJrQ?q2B+47e0IqD{O-9lwS&P@M}!s-0j& z5N2ME=B^&b$F{nwNa}yKv6f;)!2BK6 zP0QaY<&z;u9(YexG`d1MO8TrEm8Sf`oEffk>v^=a`eQhKY0kZJ%pG=GuybXGd50qfkFhi$=f5)B|GRDW=ac^)nB_<8c^v;OihrE1|6hvY`|8jw zpM_KZSXXnC(@z#HW$!F%&y==gB3T5SJ*RU|Syb#K0)_rZfxF!fVXe6=#tSSnLw8W?$gIv1e)_kRl>B%|sWEsM zU@_`w+1eL_m$%+%ZcP`s!1DeJr>;Ee54{7_n6X*B|54smTy1jhe#9S|oaOH%3WUHo z2f5p+jV2l3LW730w)PnwOkvSyBiWDQv3aJF87!(%RV+GyJQjpaHh+u>5z#Gp8ds!y z4tMp6vHIru{9U&$_I(4>mj$v@oD$X(W0Tr87f6x!wj z^A-N|)LBP9QRu8@8Z3;Ym4o!_0AG{Ti100r*SFqKf6aYGDIK?Yf1)3I<3txvY}Rb| z!0Zy>k;m*nR!$L1UkNVV3E_?nFB@*lu?Q)}y}qOooz<4I(zetE7P?DK-ypk!?1Z4vl=-|b}KCsM!l=e9H z4BHh?7AKca!tB-jfR89duCdAuMrP3sGqu|+YPx2=ViNtD-^Z~fti`k??S`3MDANUe zwy_NC%wCj_$_j56o^}ji9}(WLN=dU)5rIamipaD{yVAx$>1X@g!<-w>mu!cZxwB@d z2*1e%qR;oE%I4nF&lTph#<^fPETMqOpM!!~Wq;mt-1qTG=2uz8dB{Y5?Xr^Fd?D4e z)dD3_($BA|y*HjV(aq})Zk2M*D-u!!9p7z~u%X`_Y?0aFGK$Si1qjl!vf5&^=B2i3 z?&xM{UZ^_=xN!(VXfn!va@EAryuVpcnxOaO!Ebb4JG++BSU_u%KTc;IEeNg-(y)v(yq)F!sOE>h|j+0ywy zCg<)oweDYu;fFptq53a^-G5N${^8L5Q~7@jnNE@igjf|tTw$Tv9#PjvlC>89*Ur#l eg78d^MYgf1;rw(f)*q$-#s(HwOZ9F)`QHF}_P@gb literal 0 HcmV?d00001 From 8e9803f9ecbaeffe8258380a3fa80992c8b57c1f Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 17 Jun 2019 15:57:05 +0000 Subject: [PATCH 22/40] removing draft --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 1 - .../automations_monitoring.md | 1 - .../conclusions_and_cleanup.md | 1 - .../emr_instance_fleets.md | 1 - .../emr_uniform_groups.md | 1 - .../fleet_config_options.md | 1 - .../launching_emr_cluster-1.md | 1 - .../launching_emr_cluster-2.md | 1 - .../launching_emr_cluster-3.md | 1 - .../prerequisites_notes.md | 1 - .../right_sizing_executors.md | 1 - .../runtime_metrics.md | 1 - .../selecting_instance_types.md | 1 - .../spot_savings_summary.md | 1 - .../tracking_spot_interruptions.md | 1 - .../verifying_results.md | 1 - .../visualizing_costs.md | 1 - 17 files changed, 17 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index a353d30c..6f556bf4 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -2,7 +2,6 @@ title: "Running Spark apps with EMR on Spot Instances" date: 2019-01-24T09:05:54Z weight: 60 -draft: true pre: "⁃ " --- diff --git a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md index 2f835496..b29875d8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md @@ -1,7 +1,6 @@ --- title: "Automations and monitoring" weight: 110 -draft: true --- When adopting EMR into your analytics flows and data processing pipelines, you will want to launch EMR clusters and run jobs in a programmatic manner. There are many ways to do so with AWS SDKs that can run in different environments like Lambda Functions, invoked by AWS Data Pipeline or AWS Step Functions, with third party tools like Apache Airflow, and more. \ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 0d69df3d..0af270a8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -1,7 +1,6 @@ --- title: "Conclusions and cleanup" weight: 150 -draft: true --- **Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index cf7508f8..03e14768 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -1,7 +1,6 @@ --- title: "EMR Instance Fleets" weight: 30 -draft: true --- With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 795def87..722e81bd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -1,7 +1,6 @@ --- title: "EMR Uniform Groups" weight: 20 -draft: true --- When using the EMR console to create a cluster via the quick settings, default advanced settings, or with the AWS CLI - Amazon EMR will provision your EMR cluster with a configuration option called "Uniform Instance Groups". diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index df0384b8..f77e3d7f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -1,7 +1,6 @@ --- title: "Instance Fleet configuration options" weight: 85 -draft: true --- While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index ff5972e3..9815458f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Step 1" weight: 60 -draft: true --- In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index e1e4908b..5b9fa97f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Step 2" weight: 70 -draft: true --- Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC where you want to run the cluster, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md index c75d1a9c..f5e4d283 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-3.md @@ -1,7 +1,6 @@ --- title: "Launch a cluster - Steps 3&4" weight: 80 -draft: true --- Under "**Tags**", tag your instance with a recognizable Name tag so that you'll be able to see it later in the cost reports. For example:\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index 6441516c..a86c236a 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -1,7 +1,6 @@ --- title: "Prerequisites and initial steps" weight: 10 -draft: true --- #### General requirements and notes:\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 043aec6e..8b33ebd6 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -1,7 +1,6 @@ --- title: "Right sizing Spark executors" weight: 40 -draft: true --- Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md index 1de1591c..6baffb1d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md @@ -1,7 +1,6 @@ --- title: "Looking at run time metrics" weight: 120 -draft: true --- In this section we will look at the utilization of our instances while the job is being run. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 967c8dfd..a72fabb7 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -1,7 +1,6 @@ --- title: "Selecting instance types" weight: 50 -draft: true --- Let's use the data points we have in order to select the EC2 Instance Types that will be used in our EMR cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index da658e88..8e50d12f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -1,7 +1,6 @@ --- title: "Spot savings summary" weight: 90 -draft: true --- When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index 3182f572..c39e8e9c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -1,7 +1,6 @@ --- title: "Tracking Spot interruptions" weight: 100 -draft: true --- Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index 7e56363c..7258e009 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -1,7 +1,6 @@ --- title: "Verifying the app's results" weight: 140 -draft: true --- In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index 0f79768d..b9fbc86f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -1,7 +1,6 @@ --- title: "Visualizing costs" weight: 130 -draft: true --- In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. From 42e32d63e5a321dadf4da58607e38d0984e47fd7 Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 18 Jun 2019 08:56:00 +0000 Subject: [PATCH 23/40] small comment on right sizing execs --- .../right_sizing_executors.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 8b33ebd6..1d8ef2ef 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -5,6 +5,10 @@ weight: 40 Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +{{% notice note %}} +**Remember!** you might be able to achieve greater utilization and optimization when using a single EC2 instance type, but when adopting Spot Instances the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. +{{% /notice %}} + Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=90G —executor-cores=15**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4" From ffd4ebcd8eb1c65bf2c8760507d7c619d396d9f0 Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 18 Jun 2019 18:13:35 +0000 Subject: [PATCH 24/40] submitting workshop for review --- .../_index.md | 4 +- .../conclusions_and_cleanup.md | 2 +- .../emr_instance_fleets.md | 4 ++ .../emr_uniform_groups.md | 8 +++- .../examining_cluster.md | 45 ++++++++++++++++++ .../fleet_config_options.md | 12 +++-- .../launching_emr_cluster-1.md | 16 ++++--- .../launching_emr_cluster-2.md | 2 +- .../right_sizing_executors.md | 13 ++--- .../runtime_metrics.md | 6 --- .../selecting_instance_types.md | 3 +- .../spot_savings_summary.md | 2 +- .../tracking_spot_interruptions.md | 10 ++-- .../verifying_results.md | 29 +++++------ .../visualizing_costs.md | 21 ++++++-- .../costexplorer1.png | Bin 0 -> 31710 bytes 16 files changed, 121 insertions(+), 56 deletions(-) create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md delete mode 100644 content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md create mode 100644 static/images/running-emr-spark-apps-on-spot/costexplorer1.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 6f556bf4..86cc426a 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -5,7 +5,7 @@ weight: 60 pre: "⁃ " --- -## This workshop is still in draft! ping ranshein@amazon.com for any concerns. +## This workshop is still under construction. ping ranshein@amazon.com if you have any concerns. ## Overview @@ -28,4 +28,4 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in details during this workshop. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 0af270a8..84e98dcd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -3,7 +3,7 @@ title: "Conclusions and cleanup" weight: 150 --- -**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. +**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, you verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances. #### Cleanup diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index 03e14768..76e64bb5 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -5,6 +5,10 @@ weight: 30 With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. +{{% notice info %}} +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets. +{{% /notice %}} + **When Amazon EMR launches the cluster, it looks across those subnets to find the instances and purchasing options you specify, and will select the Spot Instances with the lowest chance of getting interrupted, for the lowest cost.** diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 722e81bd..55ceef30 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -7,6 +7,10 @@ When using the EMR console to create a cluster via the quick settings, default a With the Uniform Groups configuration option, you select the Availability Zone in which you want to launch your EMR cluster, and one instance type per group (Master, Core, and optional multiple Task groups). -When adopting Spot Instances into your workload, it's required to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. +{{% notice info %}} +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to learn more about Master, Core and Task node types in EMR. +{{% /notice %}} -For that reason, we will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file +When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. + +We will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md new file mode 100644 index 00000000..38f9d167 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -0,0 +1,45 @@ +--- +title: "Examining the cluster" +weight: 95 +--- + +In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. + +{{% notice note %}} +Do not expect to see full utilization of vCPUs and Memory on the EC2 instances, the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +{{% /notice %}} + +### Using CloudWatch Metrics +EMR emits several useful metrics to CloudWatch metrics. You can use the AWS Management Console to look at the metrics in two ways:\ +1. In the EMR console, under the Monitoring tab in your Cluster's page\ +2. By browsing to the CloudWatch service, and under Metrics, searching for the name of your cluster (copy it from the EMR Management Conosle) and clicking **EMR > Job Flow Metrics** + +Some notable metrics: + +* AppsRunning - you should see 1 since we only submitted one step to the cluster.\ +* ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ +* MemoryAllocatedMB * MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ + +### Using Ganglia and YARN ResourceManager +The recommended approach to connect to the web interfaces running on our EMR cluster is to use SSH tunneling. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) to learn more about connecting to EMR interfaces.\ +For the purpose of this workshop, and since we started our EMR cluster in a VPC public subnet, we can allow access in the EC2 Security Group in order to reach the TCP ports on which the web interfaces are listening on. + +{{% notice warning %}} +Normally you would not run EMR in a public subnet and open TCP access to the master instance, this is just for educational purposes. +{{% /notice %}} + +To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ +1. In your EMR cluster page, in the AWS Management Console, go to the Summary tab\ +2. Click on the ID of the security under **Security groups for Master**\ +3. Check the Security Group with the name **ElasticMapReduce-master**\ +4. In the lower pane, click the **Inbound tab** and click the Edit button\ +5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ +6. Click **Save**. + +Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ +1. Click **Ganglia** to open the web interface.\ +2. Have a look around. Take notice of the heatmap (**Server Load Distribution**). Notable graphs are:\ +* **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ +* **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ +3. Go back to the Summary page and click the **Resource Manager** link.\ +4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index f77e3d7f..3af88678 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -1,5 +1,5 @@ --- -title: "Instance Fleet configuration options" +title: "Fleet configuration options" weight: 85 --- @@ -11,5 +11,11 @@ While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units -This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) are 64, then 16 * r4.xlarge instances will be launched by EMR for our Core fleet. -If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. \ No newline at end of file +This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 80, then 20 * r4.xlarge instances will be launched by EMR for our Core Instance Fleet. +If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. + +#### Defined duration +This option will allow you run your EMR Instance Fleet on Spot Blocks, which are uninterrupted Spot Instances, available for 1-6 hours, at a lower discount compared to Spot Instances. + +#### Provisioning timeout +You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the Fleet when looking to use Spot Instances, before you look into failing over to On-Demand. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 9815458f..1aff90ab 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,17 +3,18 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes the application we submit to it, and will run solely on Spot Instances. The application is a simple word count that will run against a public data set of Amazon product reviews. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. To launch the cluster, follow these steps:\ 1. [Open the EMR console] (https://console.aws.amazon.com/elasticmapreduce/home) in the region where you are looking to launch your cluster.\ 1. Click "**Create Cluster**"\ 1. Click "**Go to advanced options**"\ -1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark**.\ -1. Under "**Add steps (Optional)**" -> Step type drop down menu select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -**Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: *--executor-memory 18G --executor-cores 4*\ (make sure you have two '-' chars) -**Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: *aws s3 cp \ s3://\* +1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark** and **Ganglia** (we will use it later to monitor our cluster)\ +1. Under "**Add steps (Optional)**" -> Step type drop down menu, select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ + +* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: **--executor-memory 18G --executor-cores 4** (make sure you have two '-' signs)\ +* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** ```python import sys @@ -24,8 +25,9 @@ To launch the cluster, follow these steps:\ exit() ``` Then add the location of the file under the **Application location** field, i.e: s3://\/\\ -**Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ -**Action on failure**: Leave this on *Continue* and click **Add** to save the step. + +* **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ +* **Action on failure**: Leave this on *Continue* and click **Add** to save the step. ![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 5b9fa97f..ec53617b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -3,7 +3,7 @@ title: "Launch a cluster - Step 2" weight: 70 --- -Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC where you want to run the cluster, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. +Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC that you deployed using the CloudFormation template earlier in the workshop, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleetsnetwork.png) Let's discuss the right setup for each of our node types:\ #### **Master node**: diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 1d8ef2ef..2396e85f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -3,15 +3,15 @@ title: "Right sizing Spark executors" weight: 40 --- -Building towards the first Spark application run on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +Building towards the running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. {{% notice note %}} -**Remember!** you might be able to achieve greater utilization and optimization when using a single EC2 instance type, but when adopting Spot Instances the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. +**Remember!** you might be able to achieve greater utilization and performance optimization when using a single EC2 instance type, but when adopting Spot Instances, the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. {{% /notice %}} -Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=90G —executor-cores=15**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. +Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. -If we keep approximately the same vCPU:Mem ratio (1:6) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4" +If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4". However, there are some more limitations in place. #### Amazon EMR default memory limits for Spark executors @@ -25,8 +25,9 @@ r5.xlarge: yarn.scheduler.maximum-allocation-mb 24576\ 2. With the Spark on YARN configuration option which was [introduced in EMR version 5.22] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew-history.html#emr-5220-whatsnew): spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting ) -So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types, as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. +So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio), as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) -Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. \ No newline at end of file +Our conclusion is that in order to use R family instances with the flexibility of also using the smallest supported instance types, we will use **"--executor-memory=18GB --executor cores=4"** for our Spark application submission. + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md b/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md deleted file mode 100644 index 6baffb1d..00000000 --- a/content/running_spark_apps_with_emr_on_spot_instances/runtime_metrics.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "Looking at run time metrics" -weight: 120 ---- - -In this section we will look at the utilization of our instances while the job is being run. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index a72fabb7..34b1698e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -3,8 +3,7 @@ title: "Selecting instance types" weight: 50 --- -Let's use the data points we have in order to select the EC2 Instance Types that will be used in our EMR cluster. - +Let's use our newly acquired knowledge around Spark executor sizing in order to select the EC2 Instance Types that will be used in our EMR cluster.\ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to read more about the different node types. We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 8e50d12f..02b1bd40 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -1,6 +1,6 @@ --- title: "Spot savings summary" -weight: 90 +weight: 96 --- When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index c39e8e9c..b01eaa7e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -6,7 +6,7 @@ weight: 100 Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} -In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions, but when we get started with EMR jobs this could be useful. +In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful. {{% /notice %}} @@ -41,11 +41,13 @@ The only way to simulate a Spot Interruption Notification is to use Spot Fleet. 1. In the AWS console, go to EC2 -> Spot Requests -> click **Request Spot Instances** 1. Leave all the settings as-is and check the box next to "**Maintain target capacity**", then at the bottom click **Launch**. This will create a Spot Fleet with one instance and you will get a Success window. 1. Still in the Spot console, click the console refresh button until your fleet has started the one instance (when capacity changes to **1 of 1**). -1. With your fleet checked, click Actions -> Modify target capacity -> Change "New target capacity" to 0, leave Terminate instances checked -> Click Submit +1. With your fleet checked, click Actions -> **Modify target capacity** -> Change "**New target capacity**" to 0, leave Terminate instances checked -> Click **Submit** 1. Within a minute or two you should receive an SNS notification from the topic you created, with a JSON event that indicates that the Spot Instance in the fleet was interrupted. -``` + +```json {"version":"0","id":"6009a9f4-cc7a-8a77-46f2-310520b31e0f","detail-type":"EC2 Spot Instance Interruption Warning","source":"aws.ec2","account":"","time":"2019-05-27T04:52:57Z","region":"eu-west-1","resources":["arn:aws:ec2:eu-west-1b:instance/i-0481ef86f172b68d7"],"detail":{"instance-id":"i-0481ef86f172b68d7","instance-action":"terminate"}} ``` -1. Go ahead and terminate the fleet request itself by checking the fleet, click actions -> Cancel Spot request -> Confirm. + +Go ahead and terminate the fleet request itself by checking the fleet, click actions -> **Cancel Spot request** -> **Confirm**. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md index 7258e009..44f80681 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/verifying_results.md @@ -6,21 +6,18 @@ weight: 140 In this section we will use Amazon Athena to run a SQL query against the results of our Spark application in order to make sure that it completed successfully. 1. In the AWS Management Console, go to the [Athena service] (https://console.aws.amazon.com/athena/home) and verify that you are in the correct region where you ran your EMR cluster. -2. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: +1. With "sampledb" selected in the left pane under the **Database** list, paste the following in the query window and hit **Run query**: -```sql -CREATE EXTERNAL TABLE EMRWorkshopresults( - words string, - count bigint) -stored as parquet -LOCATION - 's3:///results'; -``` -3. When the query completes and the result is **Query successful**, the table with your results has been created. -4. To look at some of the results, run this query: + CREATE EXTERNAL TABLE EMRWorkshopresults( + words string, + count bigint) + stored as parquet + LOCATION + 's3:///results'; -```sql -SELECT * -FROM "EMRWorkshopresults" -ORDER BY count DESC limit 100 -``` \ No newline at end of file +1. When the query completes and the result is **Query successful**, the table with your results has been created. +1. To look at some of the results, run this query: + + SELECT * + FROM "EMRWorkshopresults" + ORDER BY count DESC limit 100 diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index b9fbc86f..dd073980 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -1,6 +1,6 @@ --- title: "Visualizing costs" -weight: 130 +weight: 145 --- In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances. @@ -8,13 +8,24 @@ In this section we will use AWS Cost explorer to look at the costs of our EMR cl It will take 24-48 hours for your usage to appear in Cost Explorer, so you can plan to come back to this step later to check the costs of running the workshop. If your organization administrator has not granted you access to Billing information, then you will not be able to access Cost Explorer, but you can look at the examples provided below. {{% /notice %}} -In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=Name, Value=EMRTransientCluster1. These tags can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. +In Step 4 of the EMR cluster launch, we tagged the cluster with the following Tag: Key=**Name**, Value=**EMRTransientCluster1**. This tag can be used to identify resources in your AWS accounts, and can also be used to identify the costs associated with usage in case the tag Key has been enabled as a Cost Allocation Tag. [Click here] (https://aws.amazon.com/answers/account-management/aws-tagging-strategies/) to learn more about tagging in AWS. ### Analyzing costs with AWS Cost Explorer -[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. Get started quickly by creating custom reports (including charts and tabular data) that analyze cost and usage data, both at a high level (e.g., total costs and usage across all accounts) and for highly-specific requests (e.g., m2.2xlarge costs within account Y that are tagged “project: secretProject”). Using AWS Cost Explorer, you can dive deeper into your cost and usage data to identify trends, pinpoint cost drivers, and detect anomalies. +[AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. You can analyze cost and usage data, both at a high level (e.g. how much did I pay for EMR) and for highly-specific requests (e.g. Cost for a specific instance type in a specific account with a specific tag). Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ -2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**MyWorkshopEMRTransientCluster1**" -3. To +2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ +3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end. +4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia). +5. Group by **Usage Type** to get a breakdown of the costs + +![costexplorer](/images/running-emr-spark-apps-on-spot/costexplorer1.png) + +* EU-SpotUsage:r5.xlarge: This was the instance that I ran in the EMR Task Instance fleet and accured the biggest cost ($0.17)\ +* EU-BoxUsage:r5.xlarge: The EMR costs. [Click here] (https://aws.amazon.com/emr/pricing/) to learn more about EMR pricing. ($0.06)\ +* EU-EBS:VolumeUsage.gp2: EBS volumes that were attached to my EC2 Instances in the cluster - these got tagged automatically. ($0.03)\ +* EU-SpotUsage:r5a.xlarge & EU-SpotUsage:m4.xlarge: EC2 Spot price for the other instances in my cluster (Master and Core) ($0.02 combined)\ + +If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you Filter by Service=EMR and Group by Usage Type? \ No newline at end of file diff --git a/static/images/running-emr-spark-apps-on-spot/costexplorer1.png b/static/images/running-emr-spark-apps-on-spot/costexplorer1.png new file mode 100644 index 0000000000000000000000000000000000000000..b75fb95bd68bd7526290240901ef84d62497502e GIT binary patch literal 31710 zcmd?R2UJtr+BO zHIyv~0ck=Y2%!iegphVu*lv%W@4NS$^MC*S#~tHl3~{Zjxn_Om`#f)%d3r-%dmI0L zeh>(>?b_ALMj+5e90fs}^1$(DnxIKb~gyclIDqpx3p_7jF95%niH79H;w_(%}2x6TV+7pyzKKxcY}i z>v@3<&t14+i*Wl)wyz0+vj%~A&3vwUuQs~l`-Z91YOLz83Ak@sBNyly z@IeY?fHtgO;c^U-JnL7`UP;{g1*q_z7SGBjZE=%P|HU@}Y^LhriH+l$3-ey?V^a^( zFu}kSREDb9SL*YCEKQLt`VNFSOPd%bdDwo~0~JW4C$9ScDejB{sx3Gieb_1+bY4mZ zhvBT1(Kp2HEM`}BVLx^~8vFL6o`m~pyr8N)5Tj%PG0Ggf_J;*xXbLhu_2K@qCeTN# zO9yp;mXHVe_8$JI4LP<*fvQ-U^MY=S?;lsGhw+0ZByoQn)?7RC6t%fwtbR8Lxtp-F zc~^lk=#Vzgl|L3giCklz+q+#P0R)OustR2l7s2*_+^?S%MivCM7$SFF3|iLO4c-yj z5ON#@%G4`3EQ#|G`StA-*16UE8OqdDZ`JcIM8kfpjFTn`4ae89>*UQ)Ff@mhm^*s1 zNh7GTa3kndY3td?a=2yk+#9sPI5nhQSrd}R8F!yt(Cc_^U9mn?I)9-C2gezP(DQ=5 z()t>vN_rC!2HV8L8OQr|(Nmn7-bBtF^*nN-1EHGLCm-&V55Wx{%cogkae*&ZE;Va? zoN&F#s8C5TkYxnY)Zct|n*na;``qVV;OWTiR$g-FWB#%q%cBHw@e|&rCg*&A{6Yh-sfi%}#}v zMMh7~ORX=`mx=usMQqln6N0o8b7fiBdC3AxV%cQ^L+|t1+}iZdco$M5tPRdgnL)AJ zyG+2t&k<`3k5Vw3Ix~eJEto+##MSCGeJT~QS~W9;T})fO#ov%eKTGe%ZpmP?!9`)s zrIZDfV@dixyTNWx59la5-3DH<_EdS0Ifu^e8o|;F*Pi0R`7uF>o?@qBqAs!0&2R%kj|N-``(ci3S+N{Ze3C~E1x9P&nlrVM znmudh`n~iQxT0*EVs+_Ty$&{A%)`}9wpK4^g{e@iP>)Au4Iw zI{R^Ki`=uvQTz&xiB_Dqr$jy}%l4Gin!j-v{7Nu$HLN04<&oq{KR)rPv7J6bX|SauE)8H^54^UE}#& zQ63hDrXM=?#PvAhpJ0zM1TtU4WCRo_B1>`aj=W06mSN^$$v$-^L#LR`{9w^3iD=pt zsTFrktS2gGGRi`E(Z=wttO9mf2Rl*mTHfuT^DUXbreRW3e;Q1nI$r9~<{UJfYRzi) zATYLqKrUv;*3s6!?|Mcf&hJiLw+4$$S8jJg`}V?gPqQ5xIDG>Xw_cFrT=QRna#`%e zrR*9TIB{W6%dAMWAAMs<&>&D1vv-eH&rpcrOO%a%cU5gVEa5~K_%(<^{I)xTo*4(9 z7R27H?y-IngjuWQXbGOG``Y6Eu|^TPwNY2$**54B+~Fy zCXp3EWyes;Ryz8>QRO3|3+fxqD@b*x8U_P~=z)lZh2+7>@lcB%Z!O3%*A>buvY+|X z?W^qMy%xyKF?Ja&FRTR5)qv62Ag^nL!>2l^_KrM#HWG}F4AujTeny+Ecc#j#y z?Z5r9;G4e?_TA+#9!)1Ya#@)s?0OyLQ6j&Bh<^iT0KyECjm+cK6;e97(!9*88@Jg* ztReH6;W^DZ-e`0 zml%CXO@7(=tT=J%Y7#_)tvs$kIsG}!dj2#BR5)<mhod?21vp@x}Etr-@&nkaXfq{DBQvg)dHDc4w4)-r3`wDVw|;RcifqtHdxe z!qrQv?R9DGQYh9+`DmTPi&efkquCvH1CRKUR9KCqdLO4HV}dSbJOW7g&L} zxJa^L^WDSrM9vb#6ihV+3$0j+(%fn3XWbynTHPnWJK8C&nN>R@7ev4!-QGl?4^}bB zo?Cs(x^2R+V<-p)dC~SmRw!?rK`-yxesWiGtUOM=w+i+5|H`;Qv+&#AZ zkWLR7(mxGb6u_v6&B=d1(^<-!7lf{SviE9H=czSwAF)9UOsCldYXgVR5#T3hW7OtH z`&2G*6+QFZGnBhbhw7h9t3R(bUrb9RP3g+%h)s1AN32WnTM)TQ$*BOFLbglnGUcp_ zH#-}qk<^Y^U-H>XAe>AiP}HZHjYy^e3g%-3&K_j!gs-vlmn z*XqQY+oa0Ll*<}Vq4NttCqH7^n9CGwe@YMEndW29krh-17usoDVxwY%ct z=$J=uz1VzkK9>KHchKbrJF}Bbk-gK_5_bNl(i)%GjGmh;DkbDg5$sP`MIQ9REN9Bo z%^R971G{pW0{u9paa>(^ey{yhnwmj7uWpcO#ko+g1+K^r^6Fye4g7;+A&D*IK;Mp+ zkcAqLhO=QzupU+8K_%%-fo0otv*R2=>e8fH+v;wY0muVghF#BY7bg2d?Zs9*j>bvq z=g(Fl&r@QkpOX=gU{^!=E)ky%2##|K-LcOcS@o_yLsEa@`>7Yq57=yoZMg>M6vqwwB?8+VzT0<;@j{|iFr@%P8m1J zC0i}QVzIMPlTW|7Ivs89ig{j_4se3`{xnZOSew8*R+89VOKi_sE)6$blg2L}@HqfmC>$`&+GutYoC+TSA zw%hsxKJ5H5VQ55z%!Mo#SiwHDl4Ir)Ivz87V(rMZA3|(7@jav60|Jn=`+LdQZ1XGn z3u+G2eC8c!OU*n=YvCt+gIhf0fNXm4lEIBh;_8{>Mv{^39#&RTLiq9mBdIvc#p(R? zhi!>geP+taX;>?4+^#{dZ$`6k`4woNTfX<`*(-Zw@=X#I0X`^7?Akzo`}H-7?WtNP zpaTMh=CjgoOR3h8|>FWSl)&5^3t7d}G(Y2L|4)IrypHZ8F=c!y)x@I>Ah$8OnN5#hg zU6pj;>!(4m1qGZrY(9pb@16|4y5%rAf>B%3RmWiJFwcF{C;Z^dY5?s)f?lf2sJ*4J6O`r@WDW-Ci_}%aF&7U~}X;yK=pf!_0akRQxK^O%ka~ za%iFsNaE42CFO7h^1)-70`DAStatfXXhfiSpl-iZeb9@C6j9J@+98zQUhP+nn!+LJ zwK?)0yW1fehGORT;^hsMgdi^>A=z=n$Mjr(1~2(db--xd4ZftscwKDj&4-Ok zxxuGSCin$1QXEv^)gUjvPp&Wo!fbeG=;KiD0zc-)^@fk7#dlA&6(;&pSKs&tzz~CS z#Q6f*ESomiUdjB=YQrrK29mg9m~nEZY%21X(b&o1Qp-0wXJFUv3R@J_-fN;xnn;(w zF*k3$i0{gk%3I>iW5^_5FbzH4bn1Rv9`9^cbyXcYOvTpE(rTLbTJ6XykN}^tWggwu z6N6#yZcI_cUXjIqfrXP(%Wa9gvh$BR-(9>9wEbi7_FAedym>;XEH#5`4yH7-<0bWN zBGBq%Jv}+yvNE`3g?#5q=Jn%H5oqzDB?lAP7ZX^43;~5YU2Qq#t%T$85FdnBZ%dyS zQ1#VJtAywHQ9gc!i&sLsHmy&t#l-|;v^)cYSaXO zLkH~LC7w`d39l|9O7V0xsN}k&ML@`y7JZ_-%*8;j!hwfmf4sl{sn28H$Y;;TN-nJT zJK>p@Ftlui96D@(bWj60mr&!9Or!%(IeiZl>0QfbPPl#Dbuqr&E?NQ?JK{QWl)vz3 z8<ihT8U*xIaGV%A@qXqGW<_paWYc6j7A*Y(^UQ zl6S6Kve2GK>1HWxTOtLdB!eryeeHlXa@s-7v#RgA*!)7sd)Uhokh-uSRC~8sr+MSH zWaYs)Xpqu+Ut4y2mI8`g#Byy1K^=2Y2Ad;^&mP}F^c99WSvgze)Gkq4@Odt(MWru} zzqIZg^L8~w#y2!BU-|U8ov>q%fzf-gCkk_~AGk^6yJaAi#erfF)?G2Yk+Vn6(aTVk zESEWs;y-2vkf2&~H3vn3O!J3l4IRu3c>0)nMYl0cJ(f;G1(V;#;vg+9w~)O?$?>sQ zI>WPWNx!k0nTAx;pX_7Mo5m_=v=z^uW%$7iX1LJ<60y%j_^u$-zSG=c+kjpk3w+`q z!&(MZ%|TqcK%u)CgrWXMl}5ivRVP(8KO23IDa?kIhVL&_(>2N_equB( zbfJUpZXY~jmq>2TjTsiUM% z>0tKC^YJP71UsgMsu^JVt%(dNv9D1B8w>BVZ!s>X_YmQlS(8r=GMP89?$Il>Bgsd{ zXwPBkF$XZ^T$Sl$z?bB=L>6a7`+7f#Gx z#IU-=n+Oo+#6e3;8HIT(wND!zGjxKLel~hn?z=6joxFKbb@1kc&YYOma@4lmye-Q1 z*9B>b`F7hV<{%4`t_Zi}yj{B; zJgZ47XYcsn%{&*~=&08B2rO=F7~qX1k{H34qQPV5Si~8xzN#6`&6uQ2&;|uCUAera zbQTZiFI-+GjM8sWU7gKWFp`b8G2hWMG!uI7Td9O55vW6T`gdu<@#F0!R-}cG-JR%? zKzDJzer5QLV8QK>x#r%$(W}H@qdFREwVc8dzlj}cKiW=Ud<~;w8@&gZq3Pd@$Wz(z zJ(Xi)Wr6o*Git=2Ot$gLLkNl?worcDhzCwfAtcN1!QI0OtQ^zz7EIs!3SGhoc;Sw{iL zd)w@|cq}k7ms$MA9IsDL*?fkTxh5iyr(_HKobD@8q01#XR2Nb17wiQCbOt z6vf+)_QhEF!$VQp;^A|Dq7IZrT|#}Riq*s~X_R5ZVe!rH(pSij_-em2Tlo0vldfU+ z6r&F4jr-qZ%I?AH%peAPvL4$dp&+issGh~C-i6lS_cYrT3rR2ym5We%%bE#;iB$w~ zDu``!9nG!fhYND)CGiIxy8YLvYqToO>|y`aM)Wt7+02xIk6g9aB^$3syHPLrOQ}wS z=b|VBF7|&h;K>Sq$1cq@Ha_xHqU8&%+i!YH!8ZQFtotP-8M{0k>C*9Y7sq-_wjZ?< zI(zsMpSfi4%dswae;gQd=wA4%AWhFCl$kMdAJL&PIobyMfI4ldf6Gi+&ve3KOOG;! z(=d}#JWH0E=W33@V5M=bK3t=|$BQxf((cAvwKqgW?oR4GtxRr}TW%QySnF%v*!ktn=mfriq@~TA4IK${`p0FO@kibXI?YER;HvS69M+19iD69lC`sLArw z5qR{n&l%#kF}V5~;cS|K^}dl&uaBarN^GxV!QON|3B*XF>vM)qQ!4t2xEL7a0rp9+ zfMb0)sV(3QPJUe;J%foI@~Jg*9yicE)j3_C;n4?@V@c=oyE2$6Rd!}5vjKW$XWckV zWn{oH=qs}{7QV?RQ^R*^2cT`^R&F7aAwn*(*NV62)!69+Kgh6UH z{qZfU=v2-8DO`_@Y2KR?mqm0Br}3FPPWZW{7ejZWgXzcGGRB|ErFpW3s)N00PdODb zvo$RuxPJOfe1(%Q848o5%x+Ye&lUM>Us!sT%EDm5NxO9LH0w@HeN+hMPxtJz(TSt-eVURq?Ux9`l>PU9xA-9c$>NFo zRq0NZEq@k$`C9vJkmQ4nplJh`Kuk(=zo$8%AYQbieGWF5*%@MO|8eR~KdQ+`kYU)o z5ED=<8`zK88!Q&roMDG>L}A$!smMX!`6mv0!RRm3a}=F<#t8LEYqQ~?M1p&#j$1@g z!6f@=bKFW&U$eECrLWbn$BSlyntO@ee5a_t=5`b+Z4n;}-e9$+d7AQ=%aOwv{1Yem zk}NK76M;Ha%HfKOoppKc!CuQU@M-4a77Voq?6~+{JY&Qub7@iE1abEwSgA5@QKFqT9tb}7Rc6=*0r)4 z6?eCS#t^Gfts3#H1d-z0h|{o3Z;rTMZ#xhdB;Eq+x*qZ%c)*{SF}g=e-jf1ToZ44H zSE!06bY9{s_B*n*lkDMFGtS7oVr*KSZ4#5Cgbr& zM04n>`ZU1L-QYRnQ7P7c-y zgTd`Y34%dnN@$EGY2r{TVpDl@r>HVQGs3G^GjW2xyYu~umqHMHrAc00uZQNJEYHONeL+Cz{LI34O z^Q(coD#k+K?QHpPy#YBb5#39?XGb@~hllUiEo@rJ^dB4azckzJH8xmP;Ho5kf@X;6 zXgqUC*{N=@ESOo{LTN_F$O*K$wv1;_S4;Vz8<$0!`&=nU_zUAIQB2j(9(ha;?}iS$ zVQgA3ij1*kO-tgK)sLy3l!S&sVk~wl0Xs-M^=#a0*Gl%_s8cYNJ^N)l-NsR^b1>~h z%~)rTn@-?t__S6)Oz)rN^s(ZbGi!OiL8h0hH6flUOCFzBp2E(?K#B_JKCa&Fts^f) z1AW;HyUET6e!48jBg{d~>`D5YE^Su8#g9uJW2eKo5*;Xwn|Yg!tL~;X&sv8|G~IqV zeYrKuND!)3eLc4v)Kipxgz||GvR~4*A}&&%fiNASl>yW zwzxus$ETc}cmlcSrL?tJ9b?DK^}_BUtvg1tq- z9QaTIMqp4c%t6^^vfQ4m=KX?18ZEOv{Sh63IlMf0jG&4zDvI>bU-dU<;MB@J*&7(n zD--wwYK{>tM3ktEa@rS(1F0gxgXe+~-a^o^Q~^ZlIdEc$*C*|RGF5X7{c>~^0T#}3 za?$pf7dLDRmLG91?^WrFTt=N{p%z>;B2i{@{(pkS6K`v5inc5YzJwK-R8?ID0NA?C z4kE7OtJ|aaqS=hCo=J0@{Z99Hc@0F7y4m~+;<<0TCyljvNN=tejyp$Kw3-@-Hxu~! z8xZWXhAx>BO?jo{rLz|4FwMq0=dLMT;vwyB%P=+F?f|hAbche)JR-ow=JwdWMtasG zk`3eT4wy-VrMCl)D213>6$+m7HLRnSf>t4g^nE9YH?HapIMkUnai@FO@#u)`bc8GH z!JPa^vA`*Ro-Fw$qw!j@?LY|qn?RLVGw&Q5jb;bG zm$PSKPDLB&()40dV_rI@`BabD&N0Jm!shPw!M>g7sJxJR8V*(__cfs(S;-?x4iip~ zO2CA&Rso1rC5}M+XGNwMbxvAO#Kcl)Z<1H>zCprcrXnM3J=7-@O!fZV?n&vVT zc2u>WV6TR|J}v5cV9o{jdlw361W$ zQoYyNc;wr($Cn6s;%(U`$nv!9+PKEB-`kc5KaISFD~{yHb9WPZkLBgoB2T(kF9qKi zihLK*Gao#Cgm9Fd-I+fZ&1m|vAZI(8O?+40VeT7fUi)3YVR@UWPtXj&UvzuOL@S9= zd8aD6idUmM)&_zq)tggI@9c?VW*r{g3<(-^HN!wA6SXs3frbY7fughk3rhs4K`|EZVq6fTqawmN=9U6xhZme04`sxbG zGe-ZdnAgOFPzGHPEKi}xtEV?st~tu6mJ3dWfki$}oyYb~x%jxup`3!Jriy;}mAvdH zjBSF8!e0(+1qa{n9vz+0_q&xZ`~W0^#w_Q>yb*EklyH=wqDk1kPt!mzG)&#sILZ>Kz}gd&guC zhsohUU-ud!MGdV~8(u(bo(`e`@3Y)bbD3-W9{=4p!WF#B|L{Wr&4S{7@yu{F#%`c1 zf4SrGB>yvKogm;6l;lS1|MEhF|K0vyjvoudf6=%qiFbB@NSR@dnNW{l5s@tdXFAR!hRwDHFB#<|Im*6mry1ov zln9Rclx@@02uG%jyWmW1B}J@6-AlFSpp^U0{0A(2jy3gBknLyY#H$coo` zVHjFho7*hX(UZ#<^x5a@>!3(ADrh+Gl-W0ZMK16nPeOi5o2 z?RmRfR^G3k$-I9QK2(4{xkh8~7w)166NM=5x}y3#J_iAB-7;bCX~1Qf`#3h!Y_Cn6 zft)pd<+Hb?FjU|4VN}{(ZEo5Jx6AsJErkmD^s+_;u=U!68h_Sn30y$v)SznVcH=Po z#R%Tn`_UC3@-5x>Co$bV?TZ`DyK6cEFaKD4&o(~umzb0HtNBH%4hE(c>*A?y5s+p! zz2d}nai;p6MGX@-Lu7|*39qb7Lg^>IB%7?GiO$;=#@IGVx3X0Q3hn+DBhN2<<|k!P z_lhKA8@u|Q)C3gl(p619PSXh&KL`U^FfC(qOWsvN71x-XH`FhC9^@|^?#~0sghfn8 zy?LLE7C({x)(A178Qfb4+{g)(rW@&UJ_K>9)5QV(0(`!jiw5(MuJ1egIw|=zNauJo z6RoWjVj7vAy+yjc?a=H@GUB`lurM^xf# z|9ykaTrJ4$hxxAgX|DeJYxC4=h;o%5_W@L^GJC^{HaqI%sgsN!VG@}i&y_qlZv(!Q z)EBu4W1xfL+$y|ryt7)=iKvp9*}RTN72%wnbC+#LVn(9mv+p|)1~R03tru#O(a!1% zzB$^r&v9Q=ImXqFqFEryWUO4R)pKvtw2r06z2$sKDeMc>3C-H2Nx}hx9FgfQ3Rw2s z_nBes)A`EW7hTrL@@7fgMO6pQxy*>;thglUWSd&*`E&8{S@`U0_*T@{{eHJ)WJ)h5 zC-4_O+`F0k+V&@0j?IwAg~#UV$QIw+)m!UxJYKfAnBaeSGU60%qnV#YZD+hgf_&GI zr@BC)N>?;DT@F;ZZufja@MJ~g%JumbpChLkAb9$^f`yDlycfyIili4db`=8LqCV?Q zXj>u&GUH{l$M@LYk;+7k$MW^zdK@wKWOFcBwj(<`m7Bwc+^|kYKf(DtD?!d>h*25& z<~E5kd#64_PpXi&+c1l8upy`oJZ`YQIA067SmAAj*f^SawWxCc8o&L8&Wc=+&PUHP z*vQwsvT<$$N|CZF++IKv*UKV#^GNt3mkV_O5KRXoY^))WJr`RhDp5zh)OC4CjrBTjgQR5>Y@AZF zwhDYfYlgN+R0mD>>SmC77;jB&(*27HowMapVideFd40YaO?$ZZB9zDH<4})PAL8sO z4Ib0`nfoA!jX*p1{f2tDfvxsZ+t-u)r_A*Hhez9*ouL%B1b1}v!GhF5oU%iU1<8_H{s?m*W!>^L2`xP04@vZ?T( zHdClYr1yog{g{wVMaX*3&#SNTw##M!VZDNlfdHUyLi&q(@MNJ@0Sm(b%!>OF3tUOC zt7zxV^Z5r6z#aLw!*?#943I!h49kMrDuGJE#sQS%SwStFRlT-KjOnli_|8?x&_awj z>znps^n}XEauAnZsDRNxoY7fcb6+3r>@`SgriWZkZnppPJHbF~PpIb>X|n8=l0o)| z%RJ?R+E&PIoNWqb$SPpUxA+pr**gS5Af-t-;s#Q$Vqg!n2)Vn_Sh7HleE4~KmQ+JE ze<46YE;)-8Z`5ftQys09+;9-c3Visjyc&zmb#rsOJwM~i2a(LQy?x8#%sz*Hd`&>m zr*n1HSl2`kUo&B2`AM~{ai;gu+$M@81fc`f@7s-p_3wY?>;DtM7Ypot1wKG9pwj$> zbKZc{d|)y{#^R>GZEZxh+xFM({2aHC*t`so%=Pj^5c#UeW5SN-_(C6=Act=%SJ0?C zCAPXp$XFkGN;n`0eGR=AaTcuGNl1je@jFc-Jef4B(f3H}QF5_DHU|H>&*YE!*5^Xd zc`5Yf-WQD3Vp2MM~B$^cI#&YB$fdtJ{qE$l*R*5&^1FQ~{qn2G!0)>a-JuR-t zAw;%YB3+;UHp(~ymO<8YhV7-_Aqdf63ZXxU9*ySIOhQi|+mU zmS_zqm(DBAsj*zCeEsB0^C&0-)S!|g1hpu78}-*PtepLJ&}j9a)jBMYq5{9S2?SgK zU3>F#YxjG&(+35XB zz>UL$KI8CdaMA*VK*<29ive36DYZWDhDg_JG{~7BJu4GFT_>n2d+$10We7VjjeB^b zJoB4Gy(F&ish&XLE$I@Y?iPzrH+8h)bTn8x%HoQ4;87Lk3VX!H?d3>}T^z(v`p=-D z>Nege-r4WJuBkfQbc_$gZSut>Ky4?5^Ufy4L7wX_Ok)K${r-K+N6v102JA|kJV@!+ z(Qi5*i?EB(#bdZ)qQCtwiB$lZpZKj12+zfA6;*Ci?|~jbleyiH-~yw95b%!d9^lW{ zi65f20qMY1gQ*9(b=(BrDb)S7k6S?rT*0~T`D2=RaHqH_68~=Fp&AHwZ8ZjS;_g|Rjh_9_ro}~FKHcW#)DCox$RgD$T?6y3U?fI9x-Vj8X3QT3A3@>CDr!iJ zU#F^YZuTvx29wv>m3RZ`)Rlv$P-e!bhhT?*gn(bxV{kiP5}+Zs|1hA~CP8%}I~H;W zQwC?)4XXi1<#TINQfB5If>g6z>U=6wMY$O$KUZ(?6Q6kitDWUeIeXsvPCP_Z7qVx$ zCxSM3s&%z`jOgfL-jR4GvG+6wJEt;9*m3DvZ?v(&p0JJgRwk|$ z^>bgjsQd6_@n__LW+a~;v)-F2@z(yqTOCD>s67#xv(v@n*YXmo=I#0_KAm-=!&h!5 z8MdqTsZ`IlHGRHdCf1uWX-Pip-S%Oe~N$<|VmWS{6f)k7f zD*N_88~Bn;^!e`XQv=K?i`L8vcW<6Md5Jst`fLA9!|NYo$`XbeN&CJWzZ&DN56%aX z<%&Z?V?G%p8|gK+jRawJ{=!0tJd+p?X8$Q@0ygS_Tlpv`I6ZCZ?}q#2%9ym@fVhgCiZ5gL#zxm ztVcDkwFf^s3FKioG{!1bgL?4Z6Q22F8bhN3#^f`OJU+79jgWPIC)V`n>5KJJbA4GLXsko%55m(B?oUj|7Fowbd zuYmn~A;`>05v>F&W@PhO$dgvsr@^pfmwz)8#3itxKj6FR+B2-NhMIb=Slt1LQhg7k z!;fZ=vw22Jfvb82f2iy75`P}j$2ZrG$wyuOAcTK+|-*%hG7OCsw4RBOyc*dd5-H^9Om4>p2y z8_f>!TwQW^dG0ij_(%{+WXn7QWZB=4iylc2f8tor1%8EQ4L%a5+-xN-F>t>wR-YIT zdGVG@Pq@Z90I>jkO+)$m53Zxv{}IId7r1g@g-NH(tgXG=dGSOu>tOPMUv>ZgyIA}$ zdiFnSym0PZV3183xHiY;BO#fF&WQlvZAlHIY3iZT73c-z7iMfw+5RiDa8b{d#Fdiq zGfXCszX!%_y1(vT_=my)D3|8{A_@+a0!JZ~Au{LCa_KozQRt(>d@tqSb$Wgn=T-NI*H?j~Il7qaJf=rzy0aa5-C_Lg4CG1GvD(kY3GJ6M9IlFz=>w^>PK zSdl*)BI6gP3_^q)(j22Q@CDJ?`1^-Ud3=nAheimOQWa`X7I;56X5)SX8S?P`$v&on zDg*+5I^K~4Ihp{OQ)`6j4;Fgx2-7Y&-aK3hSWN$5tuE?-SMVmgviRPrQ?}30PH;@Y zZW)=Z>b7Gw=T5_fpaCQI&XwSD>A@2b*!kUsqMO4U|D)u@Uk!{g;5f%M%o+a7gqTz% zVXL%E!X{uowmt{aplc_(2Q{lfr~B|>yt7vUO+1ZOyaTub=HXcy_6#e`BdBbg%a)bBm|Dv_|5!jiOhVHn~I{stA+|YJi(8r zLpgH$R~$-s^A5^oDTim>!N8OJVz0BFGnP#nfSXF$KLf5SvNls#3Wm&v)fxvG~@XdHHJDeWFxg8@-WxtPIJ#4K`0LXD-Blw#WD|A+TU8?#=L&+?ElcL0aWRBQ)C^5 zWC8ObExiix6dyfPN=!4^G1A5v+{f+C-=qepSoG~r|Iiqik1?MyGdpWYjTVo^H{~|_ zy=^=7>)5$98zHN*tG{GdSrx%dH3F0|F17E{Yd>+k_X1b}<>!|E%T5EloK{801<>Db zA?g$b?P78y=A{%cyjws?ySOE#fs*D~8w8+JV z+6};R24#&(o*~A7mBP5Os9+B)7(h#?e>8K8f`wl^aXmzpyn7wOjk%U9z_t(@^01Ob znPW=)baBtA_A1se6Jxy#O&0 z0nXM=!h(YH#sku6q~ZW!ZE&7r@8qX~1OFtg{{(`Xy8*j$wO(hz0I@Zp67}HZucBKx zcLO=sic_PCZFc!)0z(8E507BO(G}mvf?5cdfOn6H=da9yPQR9rq`dt!;* zZ<|9M2n$R*WyVEh-@8sEKhb(X!oNDh#ov2zmjX0nlviV4Stx~|#e~T+O;m%sy^L{- zAS0QNYXNW8_7k6sw=Gm)T;CChZ^Y6`M8f2v35=>Bmyv9W2rjfzpz)uNrOi|wj% zmZErF*0;bv;9N-x7w!SOkOd*s6WEvcW#XkcU*Z+U>r_zXc>%k^UlkpXvbkdKnj0h^ ztYsLz2MEqdHu7GNvzS!5m6RpP8~7I1%3o6n2}sl^2jNL2V{DCp{~f9QzbFA$ZQ=7F zc`8qjKb3@(E^jnZlD89Elghi%QW_FprJwV0y7!+zH7K?I1X;`iUlbF6Yvmhn9wODD zgugIlhO~ka8-G0J>X9jMY%$;wudJZ1YY{1|mAeEjbK3GeO2??k)MB}MW~&80?& z@0R{(yKnt&bulklaiI?6Q-!l<=KFc`lDalAXx|3-l4PF*SqR~e(HH7}0L``r*tN;T z;Zduz#nhonkbsd?qQk_J?}K0Up{rvh$XF{@-u$8k@o09GmX0Z@Whpb`k7E8pmwiVD zg`oaLZ!3b=jY$JD9^s1j{chLhQVMw6zrgtwfEe1pJ}vO;yl3YwF9w7klaIG8(pAKl zKIQ8-)zTdeV8HG-v`zLiE^7K8=hQvj)e|L)%k#Gn>6_qn31z|0{NF_Te)^Zn?S)Pl zx~mG85&I2x(#Q1Zw_}I`$ytRn(g9b_X~i!KL7ha!AWcVPG1da)oNRoeb3%~lU_5a0 z_zRS$ugt~7$^yT*@ZMEHr#PKz8P~yIm}lPoBl=lzwl<6rXnQY>B#VpY`i_4()CNsu zQ@xAcTz_$=HZQ3SpZLc7X=Zeaf#ZceKX95k4UIp4QF?NjnMcD%zg0^dOt^lyVbV;; zfNWHdYk~B?7vP-g!YOY)4{r219z(y^Chzz^1bB4K?7;JAAUzi9obyn2(%2Pe`x`ww zu9&_lVqrSF$fs50eA6Bv75aA&FAVtvaUHUw3-K4sF9D9t-#Nnn@7<)EWpE?@lsq+c zFaH}FRWa@>--%edevr_wg7Fs$&o)}_$SF}yzbQWy;VW+9=ophz`^AWAb3Ukr{Gob*HCbd*y2WjYMkJf|I63B{RXlQpeu&~T2g)Tqvw0sb(&E1OI_!= z(W)6K+&t>$j&ZZ&z`?M8;ZWegEgf9k1D$8B?3gg-Cd?$utRGa%3I$YF!n7Y}Q&Qj> zo?~{zO>#d*k`>4Wbnrq~qcPF_(gxgCfbVs1Rpie=QC01`0|}}fMhVz^zYJ4*0(FEO4|@YTFINBJAc(2DL_bvz9b`43k3Ht6W58;)0g)F zE{|86e=lT$3@Ii9rgdbo(d=5W8W@|`EXY0e2e7uSzu665MJSMSqXgFIHjk<1VbazP z0d7nZB*6_t1CfA!tJ!X81_N?s8sKE>Z1E?{Ruai54Tm_)GWd`2?Ok8kpYel!tH|!Q zHHOV}?z8vHNgrw-s^gdQ+rZZ7>#g{ppK%sF<=J&XtWTU=v%ZqR<&x@?Zm zUheOu{3F28KURhikqucAz^(h${x%!{dH?P8|B&GQw_y?7OdV%>b1#BDi{8^c0rv9m z61$?Wj3AG)Di*UhfxH32yB=LB3@89G@C2~lABXDzoLATc438UyLH2Tw+eU5yt`xbP zn;WRf69Y2J?13{`ph&I{i`!`^bsobC>*cIq*}jkMtAdym5@Vut1L)GvcSf4K!0*zT zjVK__G7?0@e%+z~YWVr|zpGJ`-M3)IS>SkIxSBA^5a&eGHC5+8xSd>^fg4UD)228@ zYm4G+zKefMcU2&y*?)qN!1>U>=AjQb-L~@-ch_G5B<`;<0`}wMXF$H+1_j=~heu`1 zSL6b}dJ_HY94Bf~Om)tROVmA%ui78)dV#ZYt0|_$=+Q4v{Ui~68<%b0d)nbhK z06UkEIOnJNMnR`R4AMCL@xJ14V^6J;--=3jE!z`BAV@4EwIUqaW*r;+7AnIJP{wcZT+hCNDp=Y^3`U9_LZIbsMn>ahg z9rB;O=e8ki;Ee|-TKCI7FhRCbq~g@yU+xWg7F2hQN4UwA^g?rUN5BL{Ix%;(>sn3c z8jPd#)PX*I*>;xT{O)`o3YOL&_~&LR+)xi>$H?`~twPq5J=U3?TnB>PjlWlYuOM{r zs&XT>!(F{{V(ig@7v^Q1r`qJ2CzhJyJeDXg%0F1eY=3QUyiavb4seMF8fy4o61EtFSw4YP zB8)U{KoS9`^Y7!2x_?yEZD}wTi1C#~^U6kvX8^yaaphL0GG>o^ElaOPvc+2smHy${ z;9F1+kyKx$ngu#NMeL-mrdyn#oAa4thAt*`>y3xBFOZW);>$Q1rPwQ(-Kup zg3uL}0;VSxsmWS42E?ho!iFV5xs$l_qCK2cF{*?J-FfjzJK&ht2OS8;knnmybZ0K0l9peeKYE9ZQ5OK>He11gB}^IH zAKUsbJ+;2R<-6Q+91JJjTUdc|Z~k}(Q8sYQte1)30N9pJeva38CcO9cS_;!CMY)3f zQzJm+;$@80jyio7iOK(#XhUd7?;}fQTO=Lj9Dv%Io2hakJMx3sp)H5)>VI5T7z&$A zUm2Fhh3m@Uz7YJoJ2amSz5|_Z%kp8?Uge)E*ECQk5fcq}xc41x*OAMv5Tu~?ctqzC zSc7rsiWi!fmg+JAX;qrX@~Sd>f$+mW$%bTvw7k%*oIkE=#RJDR!XIt{5*$WeQ21nl z+yQ;H&2~4PdefcPhI|@oNM07!(+_3hZ%bKEnso=2TTOU|mS02rc68|k`-qSZ@g))0 z6F8nD`bRVtg&+)Y0IJEU0l~K$m-SP9Z3Q|Y(JBNjRJ&~N4%KQ$fKQMaz?W`1fqK<0aMejFIJ z0!$G*{CJs!V58aA%=bK?;h(TDLbHM2V&!9({whGPLF`4`1D#g}I=c=5xj%I+u(5vV z+JDu9zu>|2Cp-YU>Td}A2@n4<`u#GGg>yk;%9;f~Pl<2YC+gc->9skGmV#weuMMH$ zH(9;SocC+3+AE9KJsO7n+=7&cUjy{xx5;x+GlQBdw?2;CcD#SAk{2Iu(dV)6fHLcF z1eY`XHgLV>7ZCfwO0Pc{zI3WVJ5lkh8$1l~OF^l>c56ECfbB)4Z7a_6OEQx?j|0A} z-`6p##O>jhfiwN$G?*57Z+m<&LKMh4!u`H_@ya;hn01qFmAKhLaE2H=O8lmHRpKb; z0u?{tjk>uI42dM1MV%Qg5pPnyJ{ojlN*yu6ewCwHoxc<-O|ubZv=k0*Ugt|A=I!*z{8QT(ywf2{L!3PQz8Dk{ z(m`w3qiZYXx{H{(NL&l*={R~XVPJ@R*O%R0l;#AWezJ5*YrfdI2l-g28qj`_TpYG@=mAo?)2sr7Tn?$56-V;yX?0@GF z@$lWrSnq+)StsAttCbUJ+IXN9Rv=joFVS*yTS zU@vq$@w(Z3(h&me{@)m;lC?i2S4UXLmpz8-paVZ<6{pr0Z`W; zG!0N4dcc~8^4bG=qB`+A;D@fK73V#tY~eHTKdNbkoc4b@xh!Zh$6TUOc=Ix9XOi7b zk&Q4{fm}i51%+To9#W$1Nl3H=!Q&CNciJi;G6nmJv&r0GODOBB%rkAk0>lA|?+5YR z#0SW6hr8qjdyjan5nTpCeVTMD@oa#mk432yk8pTt<-uUDQ9%<=>v4w1KG(yU38EFv z^4VUPYEVQ&v8PlXqNL~rVNGV=xfHD?DrHvJojyYwV7F?>ZB*7cZ0 zf_6iC;N5gSwavL0O)N~`nxaFanKnrZ9;~EGjqoYP< z@1-tJiY^65-f0mu7|sWhH~-E^MM%qZg6{mdP5H+93zAXYSL+?I|9sMv9A&x`bKpc% zFR+|3U&zH(lZ+qGU0k7mB4FVD{oRR7S$W(KyE5Ko9RRtbSBfrpVRvHq$uL!tl>5PpKO;hbR0J$9}GOx&av|V?{ zohDJ|8)cp^ApSOm&wh|b9VNfSVOO-3V)sdtM1bqjR2)13R{oQUuM`lkfd-UE0dX3j zE$$M4zJT^i0&r67z%W$!Aw7T5r@LURJ%}6e%Ovm;;8y$s=o!nrW>4&Q&iAs!lGB|q zb7^suRK~R-_P@vO_Cx`;{}b~t<}pwgg-gQoGq`V^KPFxWv5;fWVQ8_KCGc^1;_%-x7h^F57S zaCb2zUm9?hnAPrY5QW2=lff!D3k(FH_jnkudaT!|b}n%!1X`e0*SKmQAgOB)<2q8E z0EAB?fcrTonjx-pK2`xe?JU@4@)8o~kZCn_lV0F&wLWS^z(J35e3EJRq_|)qR8D=!nPf`p7ROA_*?KtL zFg5M|)VdISvM+S!e<5fQyVgw2{Z7IyJkyfM=44PZ5Q_-OUI=8( z4G}-&X@e|+og~>}yVP^0+G zoba9UkXvso7Q@WX*%~|i9+i3!6z|8xYc`yInyhp9Av1?y)8ZFP@W@+XXL-))eo{;R zMDqcrUqzKY=7k0cq3DTPc<#+dt|aEm={x`Q$%)Uu(cRu|-v$>Kf0oNNI&nfPZf~}H z7{)nKFFoIxPntBGnrgW?p6ed-Tc=%7SG@cDofX<{rpc}?uYUwv01{VzOoooND%EKS zzt=ro`D-~OiMgQBJOjI<0BJCQ&WF`J`xa|s?{r;p)u}&lnH%?X++9)VlDFfL#|82i zMkwy-H@iKLR4B=b*{#+zcf`3^pno+#e;-b)L>OfKzmm)Ws08tpnK3K?v}XMKaovW+ zw5ZfH82IVuUe!6PEZ2q**ajBfv-3k^qN^rrucGI>1*|pJD*jqRKk5qk&Nyb+E{Oen zMSy3YUwx5xhERQ=R8HUZCqdc%lw&iDs6Q*Dl%qL+wCL4Rs;^w)R0G0D&S~A&CKgu; z0eNn4FHI(Qq(!Xzpmh30uWaNaEPaJ`hHl#d_4SYj6ckkZoie=lS|d)=-(=qvuxj8@ zDZ8oL;I|Y9e(G|&^}>%OGSO^E*QozNw&XCVpz1<+wE>=c;YPa@j|(>)t*b+H2F`Q= zolvOjk0#fjsY)|w+DEiWWw0aa6$roZvsQkxh1Iep{b27OJWmYY4pz;skUDwnI52PV(up;vuAFE?KI(w)MP^m+*T(7~Lz?2^H7*D)(7($&0C zaPPFv2@T?h{ZB<<-qp!EK~EsZ->}N!#K&cb8vj-P`%2IgM5O?9)#dHDBjo^jq%Y_B zG!wYzw@+RW zbXd)#B48Q1NRL0zqy=ZhqcW=trSLCk(jpxxX@D-Ju7f5$6+9zH7`;2Y%A|EypnWgb zkuual`~NvLz`y@rYJ&d5PE;0gQ-9|@NW*8)z8BAhY+TjLSk&T9YB!t~Lk84Z3f{RYw|WvYV8B)7f%!Wv5$ zp`+a$6!q4kq=I^&+Sea;aB8tkja!C0PVr6bu2c8uMrXwhL{}6iU_XU|BBD`m?gIu% zfUUh*BS_b~lA%~om9DkT3t{Rv|H*`Xu7}-*V~m=ybNAqRKA^uGN4CfWCamqiq=pnn z3*4BSTf;$N&(`f4(;T|EZ-$bnrm#ex;V`Q2tIn_q)z}!fe0VEiLo{LEyRz7V>OVNv zsqjvNfyLe}_yO=+vqc_RT*+|M$6l9@TrwqnWmp_tmS>TUs3(Fs*gv(isitqOQY{*8 zITrYJC~pt^NmYc=O#{oEaRN+d!F6Mo{5^6>Tld|6tv6ICRxYphsW z?bNO{^s#@HUm5|bebO`>S3QSkhYvo2MLIl(@OoAbyJzxv{$P6d^zd9a5Y35n-wujt z5A-SNLIzDTeN`CVB6(Q$!bUO#-3gW#C=VMjtkyxE&U7?ANUsnGr&cTsj~$?>2qoR} zoKs7b8yBp{#QKz}z)mI2r+1H+>S@PZKMhovH$I^jnpP)P+R#l=-K}N8>v4!zpW)3*gcr8U|P(!(X@wCI`pg$B(&>;lpo9;dG{Mh8Q`i&>jgOc|MXm3B z8!hu^L+|vohkrDRU`NzHn((z>kZ)qG7DVSi)>T(xQmXJ4;2zP@HE&t0w6%nm8*+7TFH!_e8AZ@+O2H?e`3r7RaAQI>KWBb45#G{LQ)^W8dL|W zS`z~1E08tU5$m$p0`eU7CGWud7P>t?X6BkDanZ%3Yy8XfpdbAp&B8*X^6Jpl4rS?* z{S^so5?_=Zpd$;OB;I&6nP1GrhqN);-Ehj|Y4O>KNlEHjk+{2oO^DX9R*W~kAOz@| zog`#nUZ9U~E#_ROc^%fu_uLgChFFB7S^V~I`bA4wTC_0{JANaH)CWVk`iV*qjiT9w z^nUe$66wB^6BeQ~o}sApE%ROy?6&@%3pab91kzLAj1g!d%~w!kMxS;+gU|A*GQSav zQOHzg(a@y2MSiSXUIrnP#ANHSl@bn1s&VgViH{kns9M{IrLsABUMICED+VZXU?1T> zR7#;dV9!#T8xiwP9~)7^IOq3D8hOFnLOocGl?yDSPIlGW4@&y;tNy6_<)K6^8-CO) z@{$s2e&*Bce4|}2>=O_D+AKoJV(j*D2_;`V@K~?bY(0`u;MWxlV2+{1Lmvpj_fUg` zbMEcQC+f^9JEwRkcCyZdum4i}lCmYuADg*2cgAX}=~{Wphr=h`)HvUuYq@=}U2VKK ztu%{7w6%0HcE+2tNrFBGy2DgxiaT7-`~geh%ahWmL|0#?%6-lzB4WKBwY|eY3~e9X zRKafl2G%^)vRaL0xHvZH2-6dh?{T-`Wzgx*-5EFCS@X{Yid5UL%EW!rwNYKwK4-1l zg+2LBbpVNFG+_!bb6kJ_Z8Mgg>^!Va6RqzccjqNbGU$qyWq*3R2Rxd%y?_9L2>`-3~z#(LNEEk@6k zveidf{D(N(w`A81UGv)pd;G#?P=#VPV-u(QnxhXqq*)-CZVc~lLp%CNt(=#8cuFBt z%|&R|V^9Rk`w|hR#tQMnv)Qe^*W#2U(`J2ack{%HZ%d~Z?{Dd9dd|~W%S8XVhO;{a zf!O&Q_@tr)0Turg&W^EMJx}}CM9YWAOt=(WxTmbqb_SQt5~OE&HP*FEUZE9)bZo!J z@yYuAxAO{-^1~IX)J~dKd{W*VrywD#ceRzhvhXHRSh$B@S!uOo<(2iOG;6HAJDu z68!84r-<}4EMo%G97hUvYmwXHzcf{EhNRl~m>LAC!x-eaEPeHeq@AQb$v|S#Xp&x7 zr;RHYHMG1g+nyUXm{0vypLqB+c`}_SR%yaiw2XtEE{*)O-5b1^52o)%emSC-3CGJn%D-7X%V zKTIQ^kzC7y7^9(rB8QysTw2yHaR^8n7Pojbuah=J2Eo)mCgL>bi)%NOt+f zh*q>dB1OgeDtQaX5t*tUIpLej>6H&8WOPX6OXmxQ)Qr_(f#HP5G_y(Ko(qC8D~@|J z>;@{_KZ7K7!1nk~qThV<^3VKz9SYdVN@E3#b;^MLBu{VDFMyCqlng%Pn4H9VbQhxg zxc;S-GL|y$ROeL*<>Em9A<2Asiv1l`N|TN$8EtJyi~7CdWFzVQ?i;|^(*S&j+7TMQG&|Wk30nqVKhsJQSy%!hC|aPz$CzN&FhuT&3O^9v5e5OsMdfBs;$G4`l?tS#}?)7AhW5E1gud;szYv zu>+Vqch!Pn6;&5f0LB<*S)8giiiq=0<89=y5v_qA*%^<5;+!Xrda3MlV~8x>XP`k_ zoY6MH!k{*DC2IFR{=w#WuT0+KtCq1|+O*>uIRDVPnO@h|HRroHv=d9_CzSF^VQ~WA zY)d|CMpSn+(rH4R5ji?%gsMVBt8M8)Osw?x0A&R|!3-a~srP5!!@Y!yRs9G-OIJpyPkLU}-clt-Ygt3F|+ou%C8n=c&Pmx>P}(6OsG z)>Goed*;LXBL$Xs0>Wl6BwiX#Uf(Cc0_Mr zRH>}#TcgM4$Ufh$Sh;8CtF~5w6^j@#?kb2&GLi&DbVZ#_`VF&P$+f*C6eEbbNVZpU zAdFw5u?ZyCsRGgKJu1InM7$Kkk#6p(W*$(>LIMT45g>W<6-xQj_mf&81J#g%rU>(4&DNm7}P2^Y`~%4@bGyvSoRh((k$qC^GRJM zrBwm{@VZ)KiH{h{wr~Z`)x%h{jct}`Xc$LcGNI0^waMb{Jj{+heqB`C@#4Ih9*+S% zhV5uUeH49!e!;(P>oBOMy}sUjZMt;>`xM!|UMHb6D3sP^K4W?I%`Aq%j&zl*Z>4N` zk%*v?QK49qJ2Wq`5UxubBwz}?s=ps|Q|LYoi|e}*0+pC(w@hES(0tsQ-Kbw}{m!71 zXZewvtRGmrZj^?O(B-Sfg}Q({4)QZ22NhT|&fmh=cLAz)QW9>2jV?g*7S@U6;~q2F zhPT>5mGxt^)L7wHgR%@4APRbSnnM&Onskz2Y;qoL#sV?p^2 zt=02wV2QLCn2(TgtuI6#+sb;t89LI-9TTDU#XfPl+e{l8metn@*o3$FYzEY79(z?3 zuFw`ycF&}~-mYtI->CQPn;Z%uQFs763&nsuGpcwUPxd%t<@cxuLa@vy2~;g-%fCFJ zs22ebP9-}R9+%n1P2Lf+1U%7%MVzj6l&@N0PYc%SFD(LgeAyBzjCv71-J1-mbPBQl0DASiW?uCz4ZcL*@)_f#TDW4 ziG+UIu|?OSsl%^(dgnyvpk*Iut7{?U{NJ{kCinJS5y$c{8P^WCPEz6GL3!%saT<^{ z#AHdiPoU)w-{g+91Vzr;HKu4n4mIz?$?}*ZtSB{V>e{*RGS;}pha=nQL$lP|$3%GL z#+k6FvA9WNIy8*ns$_6-P>ENhd>)CISMvO#@(1&+g!!~jGJP}WIll|VqyDN=o;MLN zs>Mb-!ma5*<2%?qhjuCxKWIzR{lX0*@w(WlVemp;6AiUE=(!@JlloL#+^(k7uP&x&1aa? zV!RBCA!n$1ici~vXE4dFKo0h3{XC~r0Nv0!({&HsPDr)pSLoO4 zlKbWDDL|}2Iwe4bX@A4V__F;CIad$kpCKpW^DoEFHH{YGzxgTdvCW^Foa3EpwDkzT zJG|3xo~pu$c5-mtMau}mE)ZJAFTN;v|07TB9Jq+#A-h;YvL<3f{rj#%_ogyxb^Aiu z1D?f4*HGk1>fW{|YyV9f35?hD+uK+Zpl`#>$?$jGU9qasX_d=?pkznWY1LL#>~J=m zQFr&Jbg=;yj58srl(KHg{UyKDGLI$L!(uri#x8YGRe66Si-58$ePq86Kgfz<-5099 zFeojAgn*0D0Mce2d0^0W%kwYxUv|0dNhZI*m4kw(=z1@xhui8c!klb~94k(aBhR^G zWq%yI_a^=IVbt;AROY(-r#BRhGrR3}RVKjkmINKCh|WD5pgYtBjvlsvP=q&1%B0K4BbP zzWc9Fj*4~2kIVac@T#xr!tbkG!Z*H5%fmXfF|^m02`UKXgq?ZUHkyhqe4<**Y|7(m z=d^j#3;E*f?bXQ3jloCNem~@)$r9FRXY>c}Q^}uj-%M>v?9p~>*T`DaUhRf+7_H9w zingsEDuf*8q*F*ne{~yImG5tdGqG_ra=8U9U3=wpqB^y?MYdpf6a6W9n(r`w_jSX8 zQS}&kExEW%$InndWwE-OLu)sO6;O`6JSI+~s!~oZHHnoI;b(q+fqCBf(h9*0LDjYT zo4TvHLGi8QcKF&wbQjP)efATb{XTD1ngHXz^>Uqrh&c#LU;&Oa%*3gk651UuZ<5oi9jAim&`@`7{aX%WeY_MYBCf-zmQ;p&M_ zLCv>(ipsy9acjDxqoe7!^8YDV8_()f3}q&2+ycgbJT5>8?Fc zB}GnWbl*^qEDn`$m%7IVjD=&o1=ANX(hMT9=CHKKJ`nFM_|}>_I-RxMlyVq^ z!~ga;j_tyui>2?}+)4t_sfc|sny&zQQP`hES?B^{QL7^bi}^RUK%^vhRbOrTO@FzR z)(H3q^E|XS0}%s0m69vV*q^a9eq8}EyU*I}8RQ(P7UOM=A%vd0?>|DKY|~hH9x6WP zwU9T>6Z*{EGY*J$DV)YhDiOsVksFVG^Et0k8iBVYyi^yz(i8Golp7ngd%jWx-Qin) zhe-mxRCXbkgiR{4mq=~9^XQ}ShIUX_^#A{C`JPk)v1 zIV#64FwY+*Q*Q2`2f0p2fOj8|K__048Ieujt*C>|%uzGdLmhJ6G^#8STPB}1cGU%! zbtHC0GI2Vjly(x)Z^PO9rY@%&Z+}ZFQ43a13W&TOfJny=;>g!hdDG(TK|785IElAAhG^~nr;qF#B# zgUBr(G4P4nDu;{qek|$bGQ=4${#!Q!DP5u z5ti8bejUXok=b)_ONhnt;N zlFlKpKT+_*bs5u4|5DMZoRKrN-c|XDvEF`VIrn%(kYUJnVP79Ci4RtOU6-!$BsTxU zjCc2O9#tpo;uSgRe%bBMIF7toZjZ#@i`WVnJt{n2-k`dmY#8&tj?Sdj!el;g)h>c) z8AaK7L{~SJ`6R6~ksd!=ppz zZ5_wImf4Y!&98k+%ue@TGnzlD9r-$Jey04{?oIQFnTS%@+0u=9KMF6kU0)D%Gu>YZ z<{dH!J>SeLTTxsA#+O<#%I|tFFkHqP=vSK`1?B^yY{btrH!n$4gf7h=<8kVyj0>gF zTE@u|x)GTtR~pin1M!#J=?IFTl@oLt5u!FZ+`)o4B#cwcWXs5|uPCwi`z`X!Ds zor;T0OJy7l-TqLT4R+6VImYCtx*Y4Li(Fj07Wr?@^I2aFsfik`S?QU0bwpeKx>acw zH)gTOq>iGQy0v(c+0bW$HMdx<0WwY+BKwzT`@fcHINXrg%N{s$YtC|q;ydGKc+qK4 zS?UjmYd7Aksfa}lY$pYZRx2076BOczsnYO{6aS+%CqxqEutf)4Xd`gpROjr7{Vlrl)!D0e818T*Hqib3Gn z2?bZct&NND4@%mS@-)sG8tB%Pb9+BR4?emu-EEM<>tfQ{f%6_~RH52e+Ft?TSRq-M z4-fE_s#9K&UeX>V&ljJMnRb##t}Hc#Vv~yGN<`f7T*17et)NU~xmvcwd9F?f)GH#F z20pF2m8+sxc*aRnAYgN}`HvA?BeGugOdyHqkm51J$P!$sXAFiy+iRkjUqsKbiLl!g zU8ZC$7K;nqzO>dF()LywD0h6iq>yUdXHV9@xk*pX&?aT2ck|D(hW=`<%{io}wIy1q zyDFo8KQ~nYcyftRM{W}GtMrqZNO-Eo?oFbX6#3{vuUGQU@7HLwPGa)gdz4XX>pE6KQlv&TtMJ>gM*H2dJAm-LVP5DG zPV-*Kh24OFUaB9Aon6E2ezPc75(o}Lf4h?C@Ye;}4YLcb8$Ogql+DfvKPcdxJmtZj zYNjmM^KNe0z)m|GUTiU(vg}_tN#}WabRf2{bxTx2z6ZKuiumGAo4fnhhdVxryEFRE zx+8nv##v6Va=74G+;4?BhO*mB0Wi=;fal>NT_TOaHAWHMw$QeALII=;TNeQ&T8o)cDYe7_I7g8;$)+BYe!gwTF-s2Zk*EM z)@{5sx1!vDLh1QZ7iN);OL4PgPt(Ibc8|I$*ggfz_^J}JY#{&c%q3I)i_brm%4MC_ z{OQsq=p*EaUSrMkvK`5*{Hkik9C&&ycYUzBD|v0vsjZy-_3pYjk8Tc*h6pMjbjEDR ziA&HAM3nAS%>6^#vBoaFs<^LWe8Izmy~-ZsOtG~5>cZkd(TEwq4SJ*ZEuXuqX&M?4 zxYs2Y@dELV{4P99P_0RegHJg?%vOUU{%vu_L;bf|+r^uE-%jwx^njDz@KSnIKy7`h z%QTA%OJpB*4a-$psUDSL6M#zb1Y*$voC*3C?SU8s*(i}iMDadJ6Pe57WNpv;u3FNdxcDb%uEwiD;`-!Y{0B9>Ku;sGg Date: Wed, 19 Jun 2019 10:33:41 +0000 Subject: [PATCH 25/40] number of executors in examining cluster --- .../examining_cluster.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index 38f9d167..fb59d441 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -42,4 +42,12 @@ Go back to the Summary tab in your EMR cluster page, and you will see links to t * **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ * **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ 3. Go back to the Summary page and click the **Resource Manager** link.\ -4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ \ No newline at end of file +4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ + + +### Number of executors in the cluster +With 80 Spot Units in the Task Instance Fleet, EMR launched either 20 * xlarge (one executor) or 10 * 2xlarge instances (2 executors), so the Task Instance Fleet provides 20 executors / containers to the cluster.\ +The Core Instance Fleet launched one xlarge instance, able to run one executor. +{{%expand "Question: Did you see more than 21 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} +Your Task / Application running on the Spark cluster was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. +{{% /expand%}} \ No newline at end of file From 4c720b01d8fad38bd2d3b2d3613de2b1036b2175 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 19 Jun 2019 14:09:04 +0000 Subject: [PATCH 26/40] revamping instance selection page --- .../selecting_instance_types.md | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 34b1698e..2b734e88 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -11,7 +11,9 @@ We determined that in order to maximize usage of R4 instance types, we will subm We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ However, at the time of writing, when looking at the EU (Ireland) region in the Spot Instance advisor, the r5.2xlarge instance type is showing an interruption rate of >20%.\ -Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. At the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge +Instead, we'll focus on instance types with lower interruption rates and suitable vCPU/Memory ratio. As an example, at the time of writing, in the EU (Ireland) region, these could be: r4.xlarge, r4.2xlarge, i3.xlarge, i3.2xlarge, r5d.xlarge + +![Spot Instance Advisor](/images/running-emr-spark-apps-on-spot/spotinstanceadvisor1.png) {{% notice note %}} Spot Instance interruption rates are dynamic, the above just provides a real world example from a specific time and would probably be different when you are performing this workshop. @@ -19,6 +21,22 @@ Spot Instance interruption rates are dynamic, the above just provides a real wor To keep our flexibility in place and be able to provide multiple instance types for our EMR cluster, we need to make sure that our executor size will be under the EMR YARN limitation that we saw in the previous step, -**Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 18 GB of RAM, and also have low Spot interruption rates (maximum 10-15%). +**Your first task**: Find and take note of 5 instance types in the region where you have created your VPC to run your EMR cluster, which will allow running executors with at least 4 vCPUs and 30+ GB of RAM, and also have low Spot interruption rates (maximum 10-15%). + +{{%expand "Click here to see a hint for the task" %}} +Instance types with sufficient Memory and vCPUs for our executor size, as well as suitable for our desired vCPU:Mem ratio, and are also under the default memory EMR limitations:\ + +**Recommended for the workshop:**\ +- r4.xlarge and larger\ +- r5.xlarge and larger\ +- r5a.xlarge and larger\ +- r5d.xlarge and larger\ +- i3.xlarge and larger\ + +**Previous generation instance types:**\ +- r3.xlarge and larger\ +- i2.xlarge and larger\ +you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take more time to complete.\ +Also note that not all instance types exist in all regions. +{{% /expand%}} -![Spot Instance Advisor](/images/running-emr-spark-apps-on-spot/spotinstanceadvisor1.png) \ No newline at end of file From bbb4f6ac2433e93721ec2b974e82d285a37ecd39 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 19 Jun 2019 18:39:22 +0000 Subject: [PATCH 27/40] notice for port 800 in examining cluster page --- .../examining_cluster.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index fb59d441..b8fdf91d 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -36,13 +36,17 @@ To allow access to your IP address to reach the EMR web interfaces via EC2 Secur 5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ 6. Click **Save**. +{{% notice note %}} +While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which is not allowed for outbound traffic on every Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) +{{% /notice %}} + Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ 1. Click **Ganglia** to open the web interface.\ 2. Have a look around. Take notice of the heatmap (**Server Load Distribution**). Notable graphs are:\ * **Cluster CPU last hour** - this will show you the CPU utilization that our Spark application consumed on our EMR cluster. you should see that utilization varied and reached around 70%.\ * **Cluster Memory last hour** - this will show you how much memory we started the cluster with, and how much Spark actually consumed.\ 3. Go back to the Summary page and click the **Resource Manager** link.\ -4. On the left pane, click **Nodes**, and in the node table, you should see the number of Containers (Spark executors) that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ +4. On the left pane, click **Nodes**, and in the node table, you should see the number of containers that each node ran. This will correspond to the ContainerAllocated metric you saw in CloudWatch.\ ### Number of executors in the cluster From 2bc1957a5a4cf7bb0ee7a1572046497e81bc2efc Mon Sep 17 00:00:00 2001 From: ranshn Date: Sat, 22 Jun 2019 12:08:47 +0000 Subject: [PATCH 28/40] modifications --- .../conclusions_and_cleanup.md | 3 +-- .../examining_cluster.md | 15 ++++++++---- .../fleet_config_options.md | 2 +- .../launching_emr_cluster-1.md | 23 ++++++++++++------ .../launching_emr_cluster-2.md | 4 +-- .../right_sizing_executors.md | 4 +-- .../visualizing_costs.md | 5 ++++ .../emrinstancefleets-task1.png | Bin 45510 -> 0 bytes .../emrinstancefleets-task2.png | Bin 0 -> 50232 bytes .../sparksubmitstep.png | Bin 15091 -> 0 bytes .../sparksubmitstep1.png | Bin 0 -> 14975 bytes 11 files changed, 36 insertions(+), 20 deletions(-) delete mode 100644 static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png create mode 100644 static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png delete mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png create mode 100644 static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index 84e98dcd..a4111f7e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -7,11 +7,10 @@ weight: 150 #### Cleanup -1. Our EMR cluster has already been terminated after the Spark application we submitted finished running. Just to be on the safe side, you can visit the EMR console and check that the cluster is in the **Terminated** state. +1. Our EMR cluster has already been terminated after the Spark application we submitted finished running. Just to be on the safe side, or if you didn't use **Auto-terminate cluster after the last step is completed** you can visit the EMR console and check that the cluster is in the **Terminated** state. If it isn't, then you can termintae it from the console. 2. Delete the VPC you deployed via CloudFormation, by going to the CloudFormation service in the AWS Management Console, selecting the VPC stack (default name is Quick-Start-VPC) and click the Delete option. Make sure that the deletion has completed successfully (this should take around 1 minute), the status of the stack will be DELETE_COMPLETE (the stack will move to the Deleted list of stacks). 3. Delete your S3 bucket from the AWS Management Console - choose the bucket from the list of buckets and hit the Delete button. This approach will also empty the bucket and delete all existing objects in the bucket. 4. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**. - #### Thank you We hope this workshop was educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR in order to optimize your costs.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index b8fdf91d..d1b77a57 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -5,8 +5,13 @@ weight: 95 In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. +### EMR Management Console +To get started, let's check that your EMR cluster and Spark application are running. +1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running. +2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. + {{% notice note %}} -Do not expect to see full utilization of vCPUs and Memory on the EC2 instances, the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very intensive and is just used for demo purposes. {{% /notice %}} ### Using CloudWatch Metrics @@ -25,7 +30,7 @@ The recommended approach to connect to the web interfaces running on our EMR clu For the purpose of this workshop, and since we started our EMR cluster in a VPC public subnet, we can allow access in the EC2 Security Group in order to reach the TCP ports on which the web interfaces are listening on. {{% notice warning %}} -Normally you would not run EMR in a public subnet and open TCP access to the master instance, this is just for educational purposes. +Normally you would not run EMR in a public subnet and open TCP access to the master instance, this faster approach is just used for the purpose of the workshop. {{% /notice %}} To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ @@ -50,8 +55,8 @@ Go back to the Summary tab in your EMR cluster page, and you will see links to t ### Number of executors in the cluster -With 80 Spot Units in the Task Instance Fleet, EMR launched either 20 * xlarge (one executor) or 10 * 2xlarge instances (2 executors), so the Task Instance Fleet provides 20 executors / containers to the cluster.\ +With 40 Spot Units in the Task Instance Fleet, EMR launched either 10 * xlarge (running one executor) or 5 * 2xlarge instances (running 2 executors), so the Task Instance Fleet provides 10 executors / containers to the cluster.\ The Core Instance Fleet launched one xlarge instance, able to run one executor. -{{%expand "Question: Did you see more than 21 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} -Your Task / Application running on the Spark cluster was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. +{{%expand "Question: Did you see more than 11 containers in CloudWatch Metrics and in YARN ResourceManager? if so, do you know why? Click to expand the answer" %}} +Your Spark application was configured to run in Cluster mode, meaning that the **Spark driver is running on the Core node**. Since it is counted as a container, this adds a container to our count, but it is not an executor. {{% /expand%}} \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 3af88678..065696a8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -3,7 +3,7 @@ title: "Fleet configuration options" weight: 85 --- -While our cluster is starting (7-8 minutes) and the job is running (4-5 minutes) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the job is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 1aff90ab..5f063e15 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -13,24 +13,31 @@ To launch the cluster, follow these steps:\ 1. Select the latest EMR release, and in the list of components, only leave **Hadoop** checked and also check **Spark** and **Ganglia** (we will use it later to monitor our cluster)\ 1. Under "**Add steps (Optional)**" -> Step type drop down menu, select "**Spark application**" and click **Configure**, then add the following details in the Add step dialog window:\ -* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Enter: **--executor-memory 18G --executor-cores 4** (make sure you have two '-' signs)\ +* **Spark-submit options**: here we will configure the memory and core count for each executor, as described in the previous section. Use these settings (make sure you have two '-' chars):\ +``` +--executor-memory 18G --executor-cores 4 +``` * **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** ```python - import sys - from pyspark.sql import SparkSession - spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() - df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") - df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) - exit() +import sys +from pyspark.sql import SparkSession +spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() +df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") +df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) +exit() ``` Then add the location of the file under the **Application location** field, i.e: s3://\/\\ * **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ * **Action on failure**: Leave this on *Continue* and click **Add** to save the step. -![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep.png) +![sparksubmit](/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png) Check the **Auto-terminate cluster after the last step is completed** option. Since we are looking to run a transient cluster just for running our Spark application, this will terminate the cluster once our submitted step (Spark Application) has completed. +{{% notice note %}} +If you are not running through the workshop in one sitting, then don't use **Auto-terminate cluster after the last step is completed**, otherwise your cluster will be terminated before you examine it, later in the workshop. +{{% /notice %}} + Click **Next** to continue setting up the EMR cluster and move from "**Step 1: Software and steps**"" to "**Step 2: Hardware**". diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index ec53617b..0ab9130b 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -22,8 +22,8 @@ Under the core node type, Click **Add / remove instance types to fleet** and sel #### **Task Instance Fleet**: Our task nodes will only run Spark executors and no HDFS DataNodes, so this is a great fit for scaling out and increasing the parallelization of our application's execution, to achieve faster execution times. Under the task node type, Click **Add / remove instance types to fleet** and select the 5 instance types you noted before as suitable for our executor size and that had suitable interruption rates in the Spot Instance Advisor.\ -Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **80 Spot units** in order to run 20 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 20 * r4.xlarge / r5.xlarge / i3.xlarge **or** 10 * r5.2xlarge / r4.2xlarge. -![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png) +Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **40 Spot units** in order to run 10 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 10 * r4.xlarge / r5.xlarge / i3.xlarge **or** 5 * r5.2xlarge / r4.2xlarge in EMR Task Instance Fleet. +![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png) click **Next** to continue to the next steps of launching your EMR cluster. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 2396e85f..9465f7fa 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -17,7 +17,7 @@ If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid g EMR by default places limits on executor sizes in two different ways, this is in order to avoid having the executor consume too much memory and interfere with the operating system and other processes running on the instance. -1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html) in the default YARN configuration options. +1. [for each instance type differently] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html#emr-hadoop-task-jvm) in the default YARN configuration options. Let's have a look at a few examples of instances that have our approximate vCPU:Mem ratio:\ r4.xlarge: yarn.scheduler.maximum-allocation-mb 23424\ r4.2xlarge: yarn.scheduler.maximum-allocation-mb 54272\ @@ -25,7 +25,7 @@ r5.xlarge: yarn.scheduler.maximum-allocation-mb 24576\ 2. With the Spark on YARN configuration option which was [introduced in EMR version 5.22] (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew-history.html#emr-5220-whatsnew): spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting ) -So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio), as they have the same vCPU:Memory ratio. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. +So we can conclude that if we decrease our executor size to ~18GB, we'll be able to use r4.xlarge and basically any of the R family instance types (or i3 that have the same vCPU:Mem ratio) as vCPU and Memory grows lineraly within family sizes. If EMR will select an r4.2xlarge instance type from the list of supported instance types that we'll provide to EMR Instance Fleets, then it will run more than 1 executor on each instance, due to Spark dynamic allocation being enabled by default. ![tags](/images/running-emr-spark-apps-on-spot/sparkmemory.png) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md index dd073980..9c37530e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md @@ -14,6 +14,11 @@ In Step 4 of the EMR cluster launch, we tagged the cluster with the following Ta ### Analyzing costs with AWS Cost Explorer [AWS Cost Explorer] (https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) has an easy-to-use interface that lets you visualize, understand, and manage your AWS costs and usage over time. You can analyze cost and usage data, both at a high level (e.g. how much did I pay for EMR) and for highly-specific requests (e.g. Cost for a specific instance type in a specific account with a specific tag). +{{% notice note %}} +If the Name tag Key was not enabled as a Cost Allocation Tag, you will not be able to filter/group according to it in Cost Explorer, but you can still gather data like cost for the EMR service, instance types, etc. +{{% /notice %}} + + Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ 2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ diff --git a/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png b/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task1.png deleted file mode 100644 index 5932f55edf06d66f76a03a4c8a6e50b11954516d..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 45510 zcma&O2UJsC*De|r1w=qZL_t9WK}0D+Kq-lUfFK>Ds1S<4OK2*+geXmsrgTCJQjCc7 zUJ|8BGYY{-m)?WaKthtUpRo{(@^2SKLp^OG3WXCEJKpSd=rQy_NyT$pu8 z*}GR_kIAJq_(y7{vvxP^w4d(ni@E)DEBg^q5+M?yy6Nn}y(>)~X%OLHspe`r!+t|) zL+iC?LF2;1p&=UWX95W#@8aBqZ+;jQy;MpnUS6*1`;f4bHN3ZvRAvP%1A(3gC!m@A z0Dsv;UgBZ@^Ol2!dcRTVuCOPAaey07nAo$fcyu=-OLQT%=g{9D6~PtN`zKWy zVbrfW_`UqOp=jSTFh`5)7|BTG{BZs-0bp2l8v~Xqo1RRC1<$_K z8C$^qxoi}g>S$`wcnnlz*RGT6@XuCxlaGqp&%niure&1RyOJ`4Fx1T-;DuRR29eRl zDpQjJMgim711@YVmu@5u5TEwdJST!;H891c=_#@609T0y5k7wv&bwCS>vRU`X{)Nia?$h7bLWk?epp3!$0n73T%v! zk+TSjd_v|-1Q=Ab_*@EJax?k_?#gC#DIZc2(wrU}wUWB8RnI`uCrM-oot55r3Rqxz zM2B%7uk!88{|jef07ZA09bZVQ6d?qt;QN9HsS2xPTE z_Snp+p>abfEnb_p7pdH;wb$r5@#{~UXlnP3C=-o8wKdDIHq8dr0gMV{11NrorJTp2^r{(2;bbCq#N^5W&oi0EJg;^f^$H#kF)GJ#oi+z3MI6ydv?R)w%3d z9;=t`upObGflD>GnMhFE_R8P(D)e!8HJGsy$zoalrwt7e!Ttb6W*c8; z`0NDGZHFI~?T(Vqt*!4C#lKLjT)BU{U=r%nNFMbFq}aayL;u>$uv)EKM^Yxd+Jk0p zEz!dQ%aSIVu;PNbH;XjMqYag0FmLN!SyAD@*K>tv?b08kHmX-rw{RarxDs8~M-1Bv~kUU{mSZ291U$iSOitsj;^7^pbmzgz%6h!3d z6GJKU#lvMg+#u)Mt#3YfvVbM-FCp%rC!1<<@8w-z$(|>iI3|~2yzm=IaE-4yE~391 z>JW`Cs*C&Qv2jvDWciF-kr&w2n788@;1Rr^yh4lf(-W#Ja%^D?=ydYI(f7^RKg8_W zm~gGNGIgdmtKjPKD(~^~Yg97YHR$@jG8A>)V|=6kWgv}Mj%9acW)80og6H z!DZFv$UN^(UrUad9Mm~Sh)~p97oG-FtUm718yhAAN}d>w7;Yf$3ACSBlSvOkZRQ_` z4=FomwNWtKhn6llECk=K{m1` z1Cin>TO%j!kE-|E3GQHjPDoWB(t}4UQ^QdD0)zm2{ed93Z}r;wOBfys-uvE_q|_C3 z6co>C_u83<@JhI4L!R8uk6ndS?@ak!HV86&WG%XIkWltWyx@y!HN#1=kbaLB)R)Bx zFKt?jWABJmcIbfjj<>Cru_$H4N_L&!Z>*5# zOC;Bw5h!PAD>rZPwfOhQO%+b7!{bH`3){BSq1R-Hj%}z}U*7ed=jw5<> z-W>aE-Ywh>F;xAo{f3bsErVo-k_y6=AEe8zWa zI^*CRjj_?D9-r|DTALIh9CWG~Eiqov!ao!-k@lWh-chjTN9~Zn6OO3o1)OH6 zH-!tLrjWh6c(2i3%}Youd(`%h9D0fJ5ntzVU4ilAb7|3HDW{Y`Bl-i-gu)nN>pFR$ z-*S8qPYMJkRw|nVWTtL>1O$qIBuLn^a~`w_QCoGUlI;9(7xJ0y)=t4xjfuqYd&H~O};9t8HF+ZgKc^t{99%p$% zxsfvM?=!o+ru)kQQgojUJ`kjH6nq4+7Jmrz+=+*9I%K^I`jk6+Kb>JZZD~t=Kh4j7 z1oV94r#R-^1Mqb3nocfUJ&s#r`xNQ^5`jf}Y!~;0-GY=p@Ufe2cAEPIxOKy*8B{ z9z_vI01PSQAzWenX&WUmvmr8;j6`$79jrv9-1LaQZJVb2YHPhY9Wn~rUT@2>UO`mECjz4T5sru$z*Sl z^yID2sfE^+HCA=s(%GjB;sfp|=Uy6k&lV+F`zyFPHhS951`CUY0!8}0H_Sd>`V;dj_V z!gO+VEk8skH~CS+^MZ`4_HG?BjMITttA95(9?an_!~eDO-C4pStg%7!;4mcYas{2! zXm#|nU@nT^t9e-yMP0{DBL0K}$r6K`@l|bOU#8Z+Uu9y03pY^YbDdNu`(x;f9wO0- z>*?cn0`YUcH6hRQ!{eTKKH5O9B=6!$HXgl>2rO(3+{R0__~{p^zldLJ%+5m>mqCP& z`#1%if(a5m4_2S@l(By?RuXc)mZ-ckLtwc?6{-I-7wYsj^h|d>>b_>J{>Tr!D-|b0 zCvH2rFA}%ieh0>%gCY&>Ys}XI-ZOUUYl6WMh{(;|bNsgaEo+L0Odq_*i3?d0Q8UB1 zLL2|fsY=8}Vr*J$wsZnsj>>ouu=Z*pd?qg(RbK4e<`F>l1IM|uaxhUqeCMCn0yLN{ zms^Lr0h?idRJw!F3Qevb8k z9F^}Bj&k*20RxankLwIU?4Hx$eMLX5_y>uZ$G~*&p&gmom9)^>pfAMtX&`b^io&UZ zW~$oyhDcllxMFg1tBdiQm$a;eTchuP!<{MXQfQAJ`7Nfze(ZpyG~>HdQq$|{K=`Hf zwB9=kxnJ+)cCmZ-z^KPgboK5B{UpB&K!d!j3bui!<6(s16nK>?kkH-0$^g2k#dMdl-!|0^cn5rPk)9CzSXNT#?#Sv2y7T4LkF>Ld z8&dFT|AF=t1)K8IU%SwdhOL`YuHx7Ae>bvQrMC}B{PH_oCgIh=?)4(0PaAHUIW(w$ zseiUf&Ut!q^kLShf&O4{vfL|Pcrdb|Rjlru=6IP^>H3f%HlMyBoe-IG@R5EZJ}IY! zb$GUk6TaykiG4Y+^>e2izqI2tV4+s>n(IdB#<0Jm@hyoK?D|f6Y+M8O0&adm4;N8Z zxAZV5r@?YEZSHXW*XkcRW&&RKyF?f*Q~&S~fVCN0m6!lok`?EK$DSoLr$?#1V94nA%uj9}cn|?4TOsZ_1QGXX=HqS#`Stl7nl%g0HJtBFbDF zLJjUvvNRfma|22ULPdWG1G}svEMS|Hyg1>a=GWSc;U4j16X8q&N_rwKC$mEM?bQ&k zxXrfQ0|H{(k(;_+a0@|c>0s`zKiLZ2JmOB{vx@rD6p7*fTQMt@TL&z4s_gm_IX0Po zIEAOXU48BT8QqVAJSWwnDyJXAHj=j7FHOYVbC=>!tW>0q_IAEg!kiO$AJV6RL67{* zdy7oHI(B#W9$q)1qGD}3DfEH@Mz^zUoAU5GUWC81P>{7Lck=y(t`bra_s;Ge%C%KQ z0T~6cY?_2wjuB@Pf2KjIs#U#?kY7iLm@$FNsK0G!hYdF59O${D2+m&*m!Ox*4L>gB z9t0&3SXAhahCv-M$2~UPI*oT~yC~rT`@Ey;!3@^1K^RSnyq8OB;C6Gk%Zp2jN>T9U z6}Jz4CV4z*-8}v4eQ{5@bd$|3_ssEv$`4Hvpsy?X5=EE0)hr}_I4;B6*#nJQf=&r{ zxi1P_7mRs<)+TgG=DPPb%-hc8_yo@jd8C^L8&08`6L#bbattonDg_-JFT+hOH1w_f zh|dr0EZi*K-pOD2XFX3)Czx?WMDNBi6{>+%{>b;Tr1HaGA(NC9HjwgWQAQl|TB}#w zZ!x1F%%7E`wUO@K>$z}zw}v1hbn{2SpYa>43^wzTpT-EIkf~k>WM3BC) zB0Ff^xuGM7O*0je&3b4Zb(J@Fp-g2er=hl2%=_x&%i2IPcSkMP)EF$efBgOO@)f%? zY;e-$Fabs2s+}3L5G0iFu*1c^2ab+T%!ls3U>%ze=cc@w|GBoQQwYjBxiO=d#3dPT zy0!7{R!NaAL_q6q_)Hpdd!!_BgJA;ivf2C?lF4|JTuR@e=H&z!V(>yLeaNq^<*F|xnY8+GUw$L6`Za!(4YOgz+RI8|sMqT_dK8tKh8v}jp!CnQZ797eKpF(P z?Wu2^ubKKJbe~)HZidanMtFo`|6$Bi)?x*Nl|n!{?B(#ZneozbVr*u-p5~po-yVHCjM*cF|glIENpEc$E|lYl`x|zII1nkwO>gRwJO2b zJ32aMP83-VZGS3y8wa@Ap9Cd33B|w+uG1&F#?xd%J3J73{`yv1(cTLwmoj%@Ht zxYic1NYRB0UFmqpcU<}Eik-~)!z+@_NPlptZM^D&b@1-afx*aOf+mp`lEWF3& z$t6sfoixcWYg>+EhfjAU*TTxx4TMlYj{n%L71eRewnPWntm{TU%%{#NC@46ppk%Ep zUyagM1jqv`c_b|k=POMAs!2+l*pU|QyL&}ctmJJ97)KYxEkUU53*&-NhSo+9NU8ve z5|grhJl6{_cSY_d3Ecnaw{R@0mP}yK&J~f*ACG#JJ=QX4jP#}zXSBW+XyGCJogZ%? z&(Zwo`dAe6h5c!l&8@n*+S(q!-L^mgAn^Cr`FWinre!BJ)n-2OCKU%du8j(t;6uir zs61R4PkFqAy+EGpy;oWBtuzVS?gB$1+KHX^Hx$m_P;psxUU*gl?4s#n9myyEe7fS*v{6jW%q*FZh&$Txir7!T%N z3iM(D-xO)R6Y|f14VPYP1buZP+Zjq2TJ!H3_^q||Mcve&jn{9y#u?cXj*^XcwL7Je z?s*~bSY&ZUyevcmU z{G9_sRDEI*O=gXA|13emU&!S$C+M1Mh;h{~RKn$qnf|VKtvbn#GPIO_WL?~G>pthw zhyOrEPS<`9tte?J#V$wv-J^}b9yNT`#sU^Gb`8%MwHzNFm~HVJ8~mu$)74+h!?%|{ zBaVLK=0BcdHSs9vlb9TNkVVg6_?zv7%V>_Y)KT-9c1ZU4xWS0jkZW$rdIJ0BS*+X4 zBd+#K)Q}zd4=oVNsDNhJS$3y)0jg182R{3_1K;<1*$PW~0vo$^xbc??(4ZNYj+H|o z-#PSOlj7F?0fsW`WKDFvlfDul#$#_d% z!mE153yYWRXD+~=2=6bR^Jl0`E`_dW;?@|JFBQ1mJsY9)+4p0(gHarR87pX8^zSxy zRQU#MV2L{?F6#|kEbuYBu}^-XX%uL}TXMx!)-D_+R{p0I5y5w*a5EXA6KY6t)p(<^ z9~Pm)YOQ6aa0%1hdMU4~NoI9(A&xV3``30egfg-sA}sS-*#Fdqs1YUK2$dE5CP9q$ zM*`|AO#9VFS8+AW4fT3@IW9s0h+d%c=l^a%!O^Dx<#$u$s+9OC#VgS$-4G9_NN$zX zEJ7w)g8j9FN{VqlhvBlifxyjuZ#|JH&$jJu}um^6Ie>@{P@>e|`m1vgBrs+;QtP;U`AOowBhRhMIIJnGFoe=D91erDNYr|Eyvq_G=J1}hwetKY7seo63ZGD6H(LC|lZ zUH$$*>+q0;Im?4gX8;U_A52KnAXh~&Ut&L9#JF1Y?s>nKa_PwnBl2!9Ui@4OJ09vI z=o+{rv~w%+WE)vZNUEmJL2b$`Fr>$Egg57`K$H}V2Ge)&AWa60Zp*IRJG2g)z%|EbyvYE8G&IrB&C zi`EEs($aT?reLK@9j^ZHZ9_zR^rcl!wHL|0W6D4T(>+>U`HWXO#XKj$>mjL5A=Lg_ zC3V&Mkb)R((d=}u<<1WYcKEELp;z$e=$H)j&-2lcKk9%dZ$#7uK!CQN@2DGxpIgz0 zdFN~@tnnf!&>|0D7FB@W&J$kG;vh+Z(FylQg0Y7MvMzqd73%k3)ML}{yG zwYC}P2}=++m;Am~fOpb=lAPchWFtfqm2n_AW%9}_;OoP0V29I3>ym;!z&^t%(xD90^5k}38WSv zkeP<@5xeW%5jh@eN`bXYTjH1utS<*V2f3Q8;;VP7YQdB5n@V~UX&}Rl^n?GN)5`@a z{g(k_>ywL+ClY{8gSVN#=OL7=Ya7cO?Vhvo9~Bhz%&(wnc-IdDSjT!CqO(M$BP&krtPtS95p&N+=PK13VUPlwa_YsWh~i<0aH=zDTUAN@p zy#Ek5ZZw#bSAY#k>9eiL2`Gh@NqSkb>^JrTD(CYV=34rfnj3CWiL8{Hn;9QbS+D(O zV>4B`%h!k)%C;F>NUf}b-CRfS`dbFCV{r@PeDe0zEkVZl^n3Kkm43P5*}Y%v_UZq0 zGN!%6Gg!w;Vc&N?l}^n3I#_-=$j_huOm*~zdR(j<_L64V{4d+pVFPH4Phppdr-THC zeP%N5)3_1LXwId6#G|3YD=hg%f^sCE;??hgPhBd6+fs_+6|!`9KT5%YG;yf*UY?=5 z5@#Mf58x(m>al>+E5u*j^2eUk9|I`F$nOq(>iv&XG!n$?t%nh1%9vZ?HRz(u?Z|R1 zBd#ukV{HWkSu2c>j4AOX$a&c>Ci3>#>Q?eYXz^pxOWr>mv*Vgz<3A#lC<~6i3=NIV z9+(Wqc~kWD7>X7x*P**2pI?16)W0(D-Tv;$R(_mD2dhMPSAqUsWf;1XMRy;b`;@sn z>-m|!K?E~A+_e0SG*Rw!$|7sjS7jmbvxdgY4sf=Oe}7tJW37)Gee7h|A|IH(&=E-V zYYGy(vf1U>QY?hbDJ4co4Unc6XeUee+z=`T$Y#oN(ukgLaP!Y7tajb|&c0a7^HTRn z>#GxSif5~6DM3JBzNt9iDIg90V(Q*?OurVkza>f5-98Ey$@-xl7a=nvK0Bm4BOxAL=V4kU1PJQw3OjffBbNrGKxt5~QD z%mV+5y@`cSPPIHwQvis4$TMgkVUd9^Iwv0MTi4{ssrSDdi?|v?!>-hb-(6y0%u1>R zMPV)j&irr0r~}$(R((Qj8o8abGHY2$xE0)lQE%&2Nx@k{+(Q#yi;vfuP(`Xw>zwRc zaTg9BDMLENa_?-tFT)Yhpm z)Iv^hsfR?OV~vMCfQq0mZ;BVYEcxf8)3y&I8GgS__&y%+-@^Z$nXx(9{ z6X(M03@q#V>)_qO zgzI1*&^sj8{GT81AKfxPZ0t?$uN%9k1oJh5HZ1_DW{Hn9h`^jxbGD1TdpAF%<dz3MYO1s5$ao-Bk6=NPl?hQG^j7 zj`_qkDQ{5+$uRe(y!8%ccVovoKd=Gs}gSg>m8J{a*A4`jTrP&a4{o4=X!<(Z?fM)I}hyRE#! zQH!!25%LBXJZ_s#lpO;b{23_q+3=8<0$_(gunCCFNz;N6rpAwU-%UN_cli^HOes#8`t5psw0YU1A z0z{}z)rF2VxA$`ApoK4&)m3PY^8j<*M^BRd-1>P586Vep<#FL`0;D zGC1Fh#09KTp_>p|MpysSRsoPBbAcwF{LBhcwv%K5!>8BDa|dT{JpO3hLv%hA=2a&| zi12HCChT4!hH>plT#4i~(hzKnHw`2@S3<=xuIny&vhg)&aT&a(klLByU*xk~@W-v5 z5{veg>H4!(*R44(K9pY~bPZgby{K-GUb2k+RerfvaF_cep`G5Yy9Lxs>`JA+^t0e0 z#Ex_E6Mj{E>$I5+R_4-a&IZ7(gP?l9?tN}Ji}3z~$je7Qe0U|*t9>9CIbn6vU&>>0IKK2%Gf6tvn`c^S@INN#HuF@%v5Lft4agcXWI>EwtpCiNmWIlO!YcfjGUIoyZe)98a93C<<;dTayK{Nb27}1w;EKFe72U#a`?OU#i zF>lwUZK$xLx7X9=czaPyga6;ou=6D91~na?t=Ow54+!1d0NWUuy<`v8Z;7mR3@os0 zJZ*{FuEu@erDp7H)XrvA8`dB{G*}e8xL=|al2z7_FkPIb%9eCr!2@^j8Wq4D)0neF zf>wW6$4@*Qwe${c8KXZeNoXmPKybN^Mgyv54!eU`){nWua-fQ00d)H{`j}O4WclOB z)#a?Ev$iH}ibM}Uz%PqK&rf))Kz1?K(iqqM8P~g~I@T}s962FqEavh4i^l0AQ3~eg z06};iW8w0NyZGB~Q_?$mZVB1|uyIUdf(B~R&@r&(F4kG&EW7ips8L1p zX~I`7O^{_J%bdTRW;fHx7wyvV@;y}ue(pZ$%3II2` zs8|g&IF2orZxR1g#d*^qx9Ma@)V=Xv)gHy6R)JScIpEX9>9vz7!Eqk7U-aGcB@>AL zAKl?(6LvTubSLo2|5K#FOq@&9!USq}M%du<(dc7?gGaePNVD7xkEB#6$jT-$6xhDD zhJ`rk`ja$H700?lp%P25>zHUNhxlp9WEWrO0jfTgZ(RjeGdO{qoA4gcp2%_O#69^Y z5{BB&H!``lkZ%Oy0@(8ZorTWQi%JMCB~3jA5R>CtoA)nV;=JBS`;#qWbJoO5MaZkY zF|pi1pDH1Pd8_Pz05p7SW))AN3@@EhhaFGt#VeOg^+p_u1x&ZQj$lHz2%#0791lvf z(>pbCTsi2b=YB*>!D&J@@4f~mPl7CNe&y)-q1q&nD5tJ=HEt7KwB~K;GFsHw*|WY? zt{%5$CjE2Lw#-6mnEJ}YZw^pVBj{UWr+NzJlrd9y?;~f!0eVd~ILN6zD}}`Q+2q{u zdtna7`Ql^?{+j`EkF6HNj=ZhD5dOj(eCM1eaz{PxxbwRS8QwD+kR#{9+m3;Q^yAJJ z6M9A=KRZiL5(2vAKmGR_1}mm62fF!!i`gG?IsN^s+%>+H?8Ay0|J@%m02J|`LuUWuN z-hT@b#wKawlZT`%gFYPs$@&9d%4#ebN#aeGCV*!^`mDeUVlRqxjs1m8AUg_^F=Syv z5D=*A1<;)KMNCHO|BEd^tG58?_ne0fUNkdZ*h^}STqe!o%FNG+z`B~?d@H|4 z`o>!nt-6MaDaGc$l=5DC9YE@qu1VATI~J;ND*!6-txNQ_(E2YXDXg41dfov4aE?R0 zf9gmE>T)lpj zV#jb)$Xq{wgo;&JZg>O=vMH$dlN_dpQ`Z~a)_mEk-W9Pw++8M8?z35Etleu?3MPde zzh1DqIx#fuOUWVyjdBZzRXZIy^cL4uT8e#2aKduR2YPkZF1fHdl%f_+mnqTQmT&6k z4sKC9wcWx6a479h?{vu$&G*Amt#DbWbpJ}p`hL3ipQ; zp!;vZ9n;#Lzt)DK#0yNmXs6G1>AGH77f%Snu?vkn7b)xg8iu0OV7af>ms6j4D7o$? z+4*{Aoftj&UIjV^i*@6zs)B;$iey08!> z5q09i{4H}5blv5>zgGWJ49W9X_O>4eU9_;j=#8*ei9(cR%r*ay%Dn3J=7ZNwA^SIg z#iS9so*XKi!VJwQU9!V;+Q&#vjv&(!rEVGht5osLLUfUzhnE`tfr{(5-loQWG3yiC z`DUIv%Tk|$7FWD0^y55iJ*nX0<}-wAx^?$g{k8ZBU;O}FJRYLcA9*WeQPTb8ad=P; zeh)qWyL1)NA0d0H@2gIZL!cz)t((sU5V||l`CoZsW@MIZxCA&=Ahup(0Xs^(2))se zu5irWbOdUGm&j;Ykl`V;KO0Xgm6q&1avF&UHgHP3<^w&>`y|-bGS}ibywWeKCxoVR zk|1^{TWCgEZGs~|Cme_o{rv1*d|*=K+CEapV9NJh?$Y-yIs7Cbqps9$QqGBT!rxgL z0{+kXzw{&?!taQ){DeK-0{v4nvnB?5TkIHTiTrou5c%N?!JoJ;Okd?`sLfRH=e4o1 z%@E@lPiSRn4zPV-VSK`o!nA>h;4|X#uqDAbA@zD{=QGOAzVJ!9&9 zFb|>nK<^6mTI$=-ZOPYjA=~VqE7{<`m{+_vlSY>l$Gl~Cx)DuG8kk7<@v9qi#b{H| zxw2`tWrKl* zUtRZ?xIT#BaI%3WAts#%B<@@O>?*hpD-8#YFKB%Iu7}^D#%u`eoG&>`NQP1rws4eW zhzzu7dLCT=};}+>M{%sH3#?$8v_m*$f4|{|9st;(~&sE1kBq#U=h4a=zp6*6(TLHfc>Z3?i}4BP_+80t zqpDwuP7_Mt|B~4%*flJbWab@qJAUY|k|W1U=<`|Oggf5}J4=(p+>PfM(9htsf8b$k zDvjLksLtuM4@U*ilEYAaE&puI-UO1zlH$IwAY{m>M=v<$2D@164u2Ew)FLhb!g6Lj z^q0@WsRb>YwLsmw4wQ=viZzRjSKp(+jV)8?_PPuBgq>xe5cV)lpu^Egw z>H4Hn|BdM_@_ttVebL@uvNXL{Bj;?yti1{UY0e1o$RXLfoWNfM9_rSje2~-IVm!fQ zrPs!PNq^~PYE6i7Qpxcjd0M>6i|bMxc{&0X?QcX0e0Y4A0NLW^k8F&SqG>`rgya=tk<;1_l`(Q(!riYbNJd;- z+VPu=y3%;9rIb9Q5Ba?~0$!j%J{UVy=#I~U#MpheYt4<5Vy&C_!ZI#*CNj0w;!NB7 zgLVH=<>bNye*bZJVsiZK$dI_Ju^ESUwE`yk#Y?PW4sUp}M!jRlZ}Huyk?IES@@r-m zIup*Aa64x{0ELIJ4K$x4R^FHBPqpZmza|dA zM}Rh1a{VEr(y{yRo;I);IPo8a(;^mq>}K--{Eb4pm6kE@kyBJPuv~!a@(MfL9cn2| zcucOpV?4Oq1a=59_SxeD!&Lxv!b4&hkFSw}FOOceAc$qk#7y;le=Sh6GxYcMU;g{+ z^)=#sjT@DITRr~ric@63Sd>GKQsApZ1yazZ{ZP6368d6Dh6Z_K~zb)AyfbY-iL~rN=SbUhc+bOSqdM z#=HOl%or-!?2Z$VIIH=uJ|sAL^ z$-27?U*@Ep1*CJIy{h`U1#a2}aII0EaN{H;poG`)U-k8`!UGg+fwnLHHwg^*yr}-K zNbi^RcbGumXPET*q{0!+ob``cgi7tZlF(Uou7mY%3qLY9!@Hsn3mZ+!l+Dz z-Pq(n0tbDr0Y{itmP(s^v=5`|k=4vlDCjw}t;K&SeZ@1_vCRYb+dnIq_HK0Smk&S8 zC=~Kp5lACWwGs{sOlhLJJmg*WZI59jIJy*PY>53_8o(ZLIqU|$-^mW8W>}kcL_xx-idOq(1}J=qofCk7jw5s!3X3x z$S&&*yJH|WfKRhJxLjVfrJH*j$^6oWEqvNn< z!kPl^U?2#!mFc%5bM0Xn#fcpRvM9^I`0;i)SB}wbxv4e{2hx_fT-y3GzVnOhD-OuM z*YS#&K>JY-clV|hdzsM#cqWI9Ya_1jr!ZlWYBb4vIZs_t-O{PQ2Jz9Z`j5~RMA^{V z%QuA|kAZ3An{nLm1HDqJ1Dm7xg%oRF@V~1-4G0LlFkeCr_!_^X184K1LYLn|?D%|IrZW9|#uM8eBA)$RAWxJK!m{LX(K33jJn_T_?_{bm+Y#74p%4a+z( zV$l0e=aGuI^JX?`wp0ABV1?)}23s+cjA^}w`7K zZGq0kXqRVkKbs}a@|yqR17D5ZnA99Z==8UofcFLS1R`(TROjv-!&%l_EY_x>2wmh* zTk=G)xz4JiP{FJ1zy(C?=E|BCSBef6>M_~0>j0f|fmte+=3#e7-r_Ng#@Uv##)&bb znmC`nZJfmZtu+o1kTJ@iK{hPH#B>*@koFjuDj%IczD$Z%XaaW>`6;*r%ZWkIw9!6& z$K<+YbUpe6&jSh}XncKgPh~{^whjw;>CVH!q!j@{!Y`miGePOXf==$Xn$uM0`7o44 zKq$hu9Wt^YRESqucT9$Pbv?t3ksT{w=N}G}*9GTV{#gQ_D~%&Nj}$}l<%pjE#iKNd z7?I`Dd4x$t0hV=RU}6R!)6|1jlM&b(j@`6?0qqhfcH{0YO$J>MKTr{1bMq&yvhUBm z6QJk$BDk2O$tvf=BY-t0(PB}c!b%J${NOlB6%+lDIq^%!Go5O?kAfrpx+1P!XXF$g z2pN$@))V0Q=3TcJ(Zj^JaxwvCi%}88&xbJH(q|D~lWz69v8fGzC)-?bhB?a^@D*MI4hbA&n*j6`sEy0KEBFh54IVD*ov|22@XS*(yhD#9KUDtlci{Yd-?cY zJZGFfO@6BKf)kh91QN*zplt2jQQ+p+W(AWr38*jr(m0-Cudtq<_*-PnEPy)=f}3rz9-Z{TomBp(+f4*3d|sGPP*zn?#*xS1 zXcY41ehSMr3)apreYl!QUC(*Zko!QeL(Z)etipAX$y&$Yz7k|m<_B8&p&vS6K2!V- z2a!CxGcrH7`OVz(X_qtL-x7$UO1w6LKk8+8f8weSfj$>!Os;I&e$lUQTlbof|IP8Kedtw ztt;(p8^ga+^vT_-02yllJpfJ@4Pqt#LGHIJ&&c4b<4RQzZp7dSnt>%UjLW5^Nwvpu z?xR?RO=Tc9Dp#oYZ@BNEom=+FNZq}#8HsiOzyyvHfKDC%;6w}oSpBTPbH58NVs1fL zz-AX2FRFr}C(!Xg()BEeFTPXeDhOnFBMdd8Y3!YFAJTXf&CJKR){0T_^bq@FP8+|lI8h7>81bNvJ$2?dAb3|AEZ*^{My#s zJ+3b=t*)khQ!%VHvPK>tx;BCH8tb_S*qPF-mF446U8^IWK?Vcg);^NlgG&h7Pu9CO zh2-&9pOT83=J36+aP_HS=N(QpMjIRL!J7PuvJ_rMBhVJ;beKR%^vwn}AYarE>M!`g zdh#J@b>c!Wse+w6RO@y4Z7bc_>heO}i_d5K>jA~J>z?CKu|Adt);TBE_dNE?@4epi zg&7~W^=3qEq02b+5!c^JXV6V97O)jZ<+P(qo$OPONOpUH`merIR(LUm%#%t}TAD=3 zq@}80%h63MSo-5Gw^TN%%Gx!=UW&B%opb4XdI;CW0nswt%(`SMp9O#hudtpp2{@y4 zP!p6DB8;l2!s8*V}nh)>&HsxN1;tMYoeEr?ih}n^J+*!!uyr?;em|>Pr846sFM%j}ZeN4}h+ad^tl~sN^b@z?g?(`M- zXP~1nn6Dbb=3__0h8Wls-z5y10ehLJjzV7zojO0SzR`VOTh)K@S20^(W z(mC%(`g@%&VI)#t=)X$KJQ2lS$Sz%}>-n2%%zo$K&Z8X}eYlk4Po;gsj0z8Nya6&U zM{XA}Nc)!KqLkVZY)Q|0+z{DOHX2%~Sp0wY{}XPL9>hZ^8!u%$Uv5EH{;@ ziy~G1Injq$Uh%L&lr_+iX}jPoHKiZa^^dp7)};pDh>T-VwmS2`P60j92go+J8$aL@ z^)0*_Z;kY3@cH7;Ifn08V4`g-5@g)Oe?)saV6#fO{SpQFBy#Wj>BkO4fH?$E&(v;} zZcD{$`8B_d^HA(geR8pS3J<2eqU3~Q`G<4{$%bSKhV}eZ`au>9X3oa-Y#f1_0TQjX z2zvX|EAbn@5oZ8L*cq=GWK&^KR^}CSo@0$;XO%+FrnqG)PTx$SUtd&S1wYtqWS*9f z2UMf^d*V-m1{8tgLb?xboD+Kb$tviQb;fyNy-}iUw}ED=gK+TI{SI`|nX`9-`uY$f z-E}{SbtU@CN=cigP))K`C>LeF7-ErQq9m<|el>FPcBv9=_3gNsuE$s}UL|EbSRo)c z@+p`oHuHTV=DaYyPBerDVT<+^f?!-%V!Q6=j=B_lBl?~mWx@p#fI$0FbD!+%!ip8T zJ=%50Y$aK)yyO^em&v%Y{P~j1`Un4ryxq-cXEiZNa^d>Hi|CT&m)YNmE6EhNy5>)d zGw{2N;GFQpl#kt^H+9Gr5NU+EL5UeQ%kPS+s}tFLVEnxrgs>15D&6^=IaGnAOFIV8 zxyb=j`Z7w>$b|liJR%QsHi3#EJA}Ce=0W;d%5914N0gPM=#t#+ zCGWP!HJRz7w1IZKuslsqq8TyZowVD3C&=FJ#J1S|-Io%@c`fcG)AjAO`q0(cQQIW_ z;uNPal>7NJ+bxgDbE7|JOxf7X8^(=t)lb9j?F9Fn4hzr18eI;o%}ZDc%rgplR86No zQpC7^lZlT@#EJ*Qq?WvEs94;v+a(-UFs%SGx%;m9lc+}-feB=uGy~Z_p>D zd0BkT6E9Il{}7B(4E1Q*rL!^+PGyWIh~VtNYz1VFN>i5=_RUY=+&<{MJb+1TAL^g> zr-TXUr?RjX{&v4|&06qO;^KG5V4&USk{~L#6g^W0 zl;5P>Na8B2hk}!^Awm^*0vD z+W%jx$$>}g)BsfY`A$o~7WnUJ6(;_hpnhEUPQy{atpdy^;s6EjGYj*~b(0Kz&T@=8 z!FZvFiMEd`DUq~cHe^R2J#6f5kDdC<7@b~?%pyZ_Cmw0i zEcBlS>^e#HQVX?XKP`s8h2C8MIkE4s|C1CqMXB_DM)XT^W2z$N4PMAQYFplIYujV~ z60s|Pi@DnAr5Ac5x0Rt2{Q>dFT2}x6QPx4S|Eedz?m9usA@9A>Zbj{xNbK*=ApfxY z?_^H;F+vt#F+}EYSaCBetxkg4Zx`9Esk^=mn3eZD>^~nt)sq~*YP?i2>_}FfojXYsT*L*VIILZK8HTvG`Kxb^}pDA z^LQxRzJFY$MG{vkg~*jgrLtuYp~X(wvrP7oE&DQ5NXnM6har1*vNKAS5F^{zm7T$4 z8N-a>dz{mC-}iN0_kG>Z^L(G*_x1Yyar>i*IcMfL&pFOx`MkG@+v!U|UFo*N%d?ap z9|P>uwmdXy9Sk_+-&_q@Ay{E`B?IQ6C?Yd}<|qE6dpG@I1)E!Cp8 z=^tM+FgI*%u%9uHYSSdKZ;8)mF`7;k%WX9XepFDb9SeeWAgrDSuYclXJ8UE8xhv&f zk^(Dt2=R2YSY3I$`8#mw;Fdx+qu;6~@oTu$Vm)$ZeV%wyw_8Es6X4r|A0B$Jjd(Od zTVK2Imd>+eGSsr~rD&09rp2>l?h7xMw4{~#_VW+`xORS9ytU=o_0iiWrCpD*^(ve( zlIdBABEO(WC}=P@%MwLntYkY%Fm4}s3lSJvfn%L>tM~NIVdtEiBV>vxKsWqBiqcpA zIDB^yMJ|FpzqTT2t)~ce!^G=QH|(aB(M|rs7eYFuAtjbm`mqs|bFQWYBeZ8j4%-|> zSEpy<3MC4fHQQ>uL1F6>p=>ek$U$L%)z}S2*A>Py-F|j@>$B!%zWu;RqQESK2mQl> zvv<2GBWsk=*Jl{9AcP|93EdXAMdow~kKsuPWDw8TmAqV!5c4ee=IfE_QtO z3xLZ2!koTGh54%1r-QC}Cl)^MeQJI}`480M_ya6;^$L?nHU4Puqq(y0Iy^M|Mo|!%nkNCBdy*@J5Vc@1ttf8|Ui+YMOZv%P7iJ<;d1G*80K--Q`hB1xTAWCDWiKREJ~i`c4>P;)p3g@b z5^@d*3pgsGUUA?ajlc(%8`vmo0uTa1y;xxC=v`biaAMV}|SyWwxMg$(W})zvI3v9no=GQ=i4xL9ts4>nx5 zy%j~~2Ffy~&bgG^4Y5wltM^hSD&*A`YzaEUZ@SzDQf2@Uw)XTTti@*75rA58;&n^q zESP}%1y@Z{=Emp74rv=561Qhxiwssy>*1~(OksyLpH0Z{TD{> zym0yP%f8c^I{8*A8l@LRQ`=1I@m~3K_ee4e$8n3zpRw`p2lsQgqTj#XFL?$YyX=2@ zHgoAI8BZ!C_LLMUwH*+5oz&?K=)2Tjkaxs&@$#i4i4dkwp#v~8<(g457jcg%`Lg>-sC5~M5j6m5IC(THE zKEWZks<7awwfsTF6}^X|jQ3BjON9a_4T0r_FL?EFzv89mXcin`7>y3~T~m!P+56JP zKqUL_$M&uk|J(b86)R2s1FuCX5RnlrkG=z^k8J)=BkuR{6Q4iSg$}O2OF;KsE1?c8 z50aqq+vU*N?+O)4=V%WC9?;$dn-|@^@xFXt44);$82>Uk90vfoi)?0w8t0Jf{MNB_ zo^dI-VDD-|;FCmo7{?)iaEdqs9+6;Sw~se z${XE2?d*A$Fo4QTc9K_<;E(Bf~y%S6-hb2DjtT0g;I0|TPgj<0THwc=& z{_#60&Ap{_Sl7{dpqdsL9bckqw%TS_$LV04}5QmHUQP5-x&e}vxsHnWCkkTlzzwW>v- zcc9i1&nbKsApV?rs3_j4rzw{Am(g1>-t#lL%kx%5h(Q+J33(FEKwuukLJNBpd$oq1 z}4$nr!GCh2C%1Rt&kzorCdQt_=^AHzJ_Fpl=YCzp___2Y3g(Nw{Hpc1G(q&By zs68(B|D04Wb}wIA(B}jK(wh8Hwx*KFdq-`St6yb_j?5t4-%zX_?wsf~95&j1^9p0- zhqX(pS+GApnd5*C2-cXuSsPm+FfEC^T&fZ= zV0pX(ED^n7FJ3#|j*V?#t-6hZ29qy)pryZxMo+-CO9O|H+hP|Prbnml;zd8ov++;4 zp4QjUtnVkv?q_j>7c9X#e2?XIh9XVaCR}EN$hf^h=<@Co4fiW~Mc-eJGRnA5J|vd9 zMOMq9v`5jLtx`=+>f{0JER1`OG;%$tjWSDFO#ca%k^_ae=6d4(NH{YbBqz|BaL2t4 zzuvTZPXp{5Kp`CP5}{d47`fck$frxsc5*%aJ>XBrX>F)ZDa)!Z4UKtim$`K(#=5({ zQ7kz%cPDEktXKCwp$j`}8qYdhJF;P`<#{DrKhi#T0DqXdw-$IsC8Ot-3)y zazW+8lBW^k{>ALpq0^Vv2%NGSNLm-K5Zt;4Ux?RH9e~t|mCj#9bNIosf6pDGY67?B zAkks}e&|au1e4rrLB$Z{9pQ@|JShoDAuq?)I?UY{%OqT019YI=7>aiNdOz-VFd)`mj)UcAn7na@H2~0&|;B*$Kra!LW zKgH64Uho@5@)>Kq1`O#Tb#aMuDinPBDtEmJtVY?`wiAoEI+~U*Nw0W-^mU%Moj|D9 z%WQUg_*Uj!EmJS8Nytm&uH3^ecTR-aG2lO8&$^|o9xqZGzwD;&pKyD3li1^zY#PR>wwnW5_t5bVd%)CheLcK$5kcP6ut4>&cVPOu!Z*aW4nAm6S&U#<2yF{W%05|k<}!bnF-xK}d4z&XUh}s3 zk!{Ia7#P1*YOsA>YdHsnH)^}3om%uA%QBw8=4 z>P4S_mQafqdH92Qyvb?2MabHLYTqNT<=x!8*1b z4pRep*&A5e^dhs0B35 zU|e9Wndb=ouB;wmDxdIuje)vdqxofr`Cdfm7&&6mTq24vj9tW%s9wE#Y)}zFP6K(#=UVIY^F|j;o5IBgi!vCrCzN7L zHIO)Z8zsak;1D_AQbcWc=^Zfn8OUmA8a-u}i?|$jH}z$`bUqx*(_-rBj1bShOSt^` zPVSdGs08QkB3Ldo4*_C7Ak@|MKbJK6rp4eweH&Y_qNv37+OXzYWiUa1fh4#znh^-v zS9lb35>JpXz+!^>%)j*z{=Ka*zq9B_XMfD_OE-t(o%*-$jW=}q&eu*xtur+ps9# z=G+N9p|Sod)F^*wR2c=L)%jq{0rh=BUlV>bsD!);UVDnW{(hZ<7l}Jn8*<2In*fTd z3F2e4N2BT7JXg;YRJof8?H4>?bXHTyW_#QAN)GM?+>bgZeOQNR)(tmx5AqTeO~o5u z2iu6nVEb0NUK|x^sB&uZFVGy4sc|p65>D9iEyDs0p@IHz&Oq#%y_NgdX29gsMKSKL z7L*Du47C;Y-Kczl4R~-NVuj!Ab+0}rPZg$V2YugZ0zRJ&e}N-{frz5ShYrZ97&L$Gd;l zY}aj@m7CRKTV~@ZyEkgx8mYyoOtAD5Dj#{7;4_f}8}xswa2(cT#O5ZW!nCIG;k{(_ z72(%cqxOrQL>l(F*_n=Zef+>QT3#rLyAx-QT|O6bi!1LXfty!>)N(PzS$AwP7uTV; zW!vYhuptlce(cJp_)wdNqCwCKVU-yVQqt2q(VV(V$X-xsPGu`~G{U-VR%k6NdlihT z`u-B%kWBXdROTRpU*eQ+3*1-+f#iiowUKHDSn)E4G4I92f0 zBRS#F%(UPy9XzYuvjR^Qud9!nNq=X2j!$)1OOPBj1)v|4c%L)z4$P#4^4= zSwdl@F^?a_{${IYk(dvAc;PM~{7n5~?bxh-b$H-MteHE?OXZ8l#sTfE(yqmU=*~V& zR(-(@aIqQ4=w`FW%O7{^puVmN70&6IEA985ggFuD%9B3eS`C~U3?9HrUnlfh60RW? zgtqKM;>AcCCuiN5dq`48$K_2TheJ}AbuVLyHaz2ZcRgZT&E#%#Pd0il2O z$txAMCGm_E9ar|%aW>F8{{3Kx;46EaBBc|5>h@@(aQu(xg-d!~rq{brT%!RxXA0Ma zZEI5W{!6E`mu40PoW56RTXESFXD6`>;#TumGmKC_@z<7~72Z`usqBj?Y@72^ve8c7 zHUZt~fgvS!d*rUl??BZ(n#?CD2iCL`J{(o;HQh2!jA|+IX^1l_%`TPr$SE#h9238O zJ383A*dZrb(QimsDwuCWUI7s;Qn~~Ali^NC12?=_cO+#-Z#Y)NfmOkRh$I*cJv3&^ z@V;VGaQe`J=PiXJR)w>|MzxHDkhN8kP#F1pyQgj8Yr)a5I&8OdA2A^YhrOAOxfn%W z32sqZg1)*n2xy>3me@aLgE>)GLv} z@qibKB?_=F})u-VrFs7$hPK!+Gp>DKIEpDr9Aebz8}yM-sQ~lDY-J6;Dq$< zUW)gAZ=&Ryw=L|+iNo-CiUfO%&aLp)?vUVYrtzRfcPJss)7ta6JJ_#d@}gk?gOrOh zKjOCWrs#HADxAk;e*^+eprX0_2xvYikBeJAqSowmnfu9w03M#%Ed|H7{!!u)(u{>R zP?32C4DFRI1xtdhiHG%rUl*Xo?qafzYpE)s8*-{84nya65iAN`a8|^-C^$tWgtK&* z2&=Asy&wmng?YeV2|R!?I~dw;#*@av{U$gh)&z*D`HVF{m&ijbg>UPwKt}sCYY=Hh zX#F>ns^%oX(f7s|qA`#yJ8&!qZxRo7;ZNeXf;h+A;M?tmmBhN;eRdkmpIG(vHG%y) zSZ!U0@XHXNNqoUpxCCVxj~GF0P3VsF2l=%G2&2T@(H?*}Nb|&#+8U3Q=3MFYs@)y( z=s}rH+hyYJkMI#@)UGL8f9>QM{109`B`kCe;tqndhz|y=r_ehRK|P7GFoy+ z{c^1Tb6Wy@H+0+ug_u}d4n^(0{Rw@65BNDn*;VD|mn2af$MWmenpVMX`4honb_(qD zu!&$-obz~m=Y`h{?)mT_r0gnhS*qlN8%Kt^x+SfjmnM^N=g=ZYasRYP9r+Xix$FJi zyK*H>?Vrpmp@juMC$z5M|AjNbCDuQZKNU=2S%ZwOuH@0;YaU;H93y(qGz1vA=MTd1 zDXvG!O?6J@r9*dRKcFjHF7gAHQLUi|GaK6O<<~q5&k)xa zLRXziZEo(Do~fj}Fg5hkz0{mn@eAeJ(|elbrLn}p@~n2HJ$+)u>Q?UV$PFdeA**S! z*guH(d1_8+f1?u8P5>71nY5&mc)0v|!fJ7kFJHtJteDVav37oU&eM?ZZH1S@J_gF|L z{3C#p)aZuhp>vS+Qd;+z65;*b!`9s^!P|E#Uw(`qallB$&58l5m+Xo@eA^gS<riVc}2h9Q=Hz~ET8sOo@8nMwFZ-X;s!7e0dDU;+MI~U zl-trX>*W=79hB95jFlY+IoG*5@`v4D%|cVP9`)|O`u>6qjKFm~JC@ilIzhH>X#3>~ z-Gvf&J=5Fw*n_|`Qf7E(*f&T9V@HnID9)8J^p5B|pMQDtpVpB+BA*v2=dB1ksW4`t8wv_uZ9)%UQcCs3vFV^z&Gru|62x{Kif9 zsz~tfZtFArKs*MX{m=@iDG)^2QYg`?LvIpos>iI``%oa8i#mQd0<7@E$QIuHF6R<{Lhlj>ctT$W&bUvH#0(Ez8Xn{`Dh*+?w~F_I z2tXB`HM|w{7Rs;PbnjwFm1XW@c1vRyV$QJqgrw{rRWdk$e33eF>6NAIX9>17ul$@> zwifZ#`Odnd`BRD7um8T6I7`JQ6aN{T>;aw{)YKP>+&~R^Dsy;Xz?eO0$L%t&O&EW+ z;-8bPK#O^y77wJr;GWKI_^?ZiPgM-Y4(wIcD6NU}D4z!8Pj91!ylm@+m^xCK?+G~_ zOY$*%M{-Sy9Q{Q}P;m4}lFq6b+xQf8)^6~|jVXL_Zmt8nYrMlJm9aO)cBIqK;&Nxi zn{cYT2@*$pX4kjT#b1X~nhP`U0vQyA&Y>?QvxxOexbm6=&RcrMkj}ch7#KZ-KqT3( zz9Wi_;n(NGrL7+q7;3(Q)=hDOP-_8bNN258^b2gu=^kraO>5ei!?Pff2cL0*B`KM{ z`FQ`sR~ONS!>S1sZoMQno=+l;^PP-D?Ea$&y9|re<@3LSmV%H*N(x zr08rs)@KK6H3jwyk0tNHt^4HRD9D0g?eTEo-RZIvOU3GZ(|X?Ol-uQykab4V=IWR4 z^6Dm%S{glTK7bo=(A@%W3k!Jt;I{PY@pffd@aZ`4;Xgn9{;ZOv9;JkHl|g}%DS;p(|calqw6vS6;D=k2GUhZ73Qk*0%IRZc?E~kyt9tAG?!%r!&Xzh;mf2{|Wzu?KCwL*Nz!v&+d<&xQ|?X&roaxFX(ev6$oZqmb8O7OSY_?>iQa3ZmimHqiw5{^ zf5q_QE&U*2pLigRq?v(~fFi(eZ#IBpzyG#$D;}1#m0sr_S#?tb*=M%*Tvd60eGy;o zv6o2m*DBTIp;Zf5b{lzAvE5}{P9KXgAFqgCSL}->2w1nr;?CY6Rxt^_?{=o_nAYB= zdVEZWOLuG+N@rl&!>x(ctX|TDl;&hz60!UUWwz>klw!N9#wXM)7-xR65~{o2epXqW zbc*ldL3UdT4kl|Rna9BL+JMK#m^<6G&0zdQCEhIYnkXo)*}fqI)`|-PVdqM4w}?;Y z%N#Xl?`f^HpIsbd@MA2_Pd%GrD;jX2eSI*I*KQLl&+fl7n^Gnl`bav_8m;01;|hfo zYk!Gh2LB@q3S22*+Ov=5TwID}rh$Wo(78YCi=6L3&oT9?wEs&V*bj+Nf$R_!^q>jb z2S_R*kbn(H4aENG3E*-habXnNs6Cknqc?a2BLvVRQ~yIP=>7lePk9-$oRZUONhMJJ zqka9G<-2~AFmJRTq&A@WumBJqh9<|r6lfx$;pc;D4_9w`UMooPT+AeIElCkIwyQjn zRwvY$ry9?~`nN!cqXFJswga-LEwD};bgqaj3E~*0LwsJa?sOPJfON;5>e?}CL6PPg zH)Nrl^CRuMZpZuY%8zUH?hpko(y6XvQm@n59CpR6wR3d>!;BD#@*iFCu&x}V+2WT} z>`#pqjJbL&y>n+X*}c{Sg%!W0>3lYqRlRgTv|}Nu+|KAuxlI zgaF}xyg?V7v{l%?aGNM8vMi@_ioc^3UYXRs3vAP#WI#e*MUC{(t<96r`983N0ImZ% zH8%yc1b8hjQ*_@hUW!Q%y|%8nq07;`zJ3}ys>l~2Y-gmvd_%GgPWKeVF%K&09sTEhbuSUt(Tdosyg%4(2xFXAcWyt)CinYYDRlBpT5EC-b0(-H)J~?X-t0 zKtBGl>K@b8j+a#uUgffs{79n{X029>Pfs4L8H#Bk$)y}CJYBVL%iIn}zA&mOrehTN zm1SyEBZapIxj8gd!zSZx*l6n`Wpn+D9=B_MOAg)54SC1!Kws%v@ugIuxr}a1$)Y>V zS8aF~u~MJOH;q*;n8$utV;Az>VfO^eWzBYRDhAxj5;hnx-~O~8YT5@NG*RB58%jVsb)2uvf2O0&LecxkBYgu76s73Mf zNQ9BQHqlV<-A?gjY?3Y_M%I^;-oLPAS(*?Rx3b9aZq+Jvu$VU~7p3lJ^i(?Q2kQnR zv}I~Udz%WtEuR77MS{xVVY&eg4W!C9!c@N4_bdF!K?PNtQm+KX^mrf9qY_S+H%vZO zj4~%ifs6#Ye#*<~>NApCVRDEJ^ov!!x!E+5keJ$5fb<)m3RzXp)j~WMZunV&y02Ox zAMe3^IsPFiPCudFW0b=yw!VLjQGTQXyWc9oiWzNwFOw(LUp6$k62^@F1^Y6Cr#U)! zA@w+k6;}3X$%T39o+sj%18n?0rZT!k4h53A5ERUR5yqrk&*7b1jAju}4uikOzhRL3 zQm2z_OI%#Fnl2=|HhcHaMl~Bmb2j~KMjU8Hc{QEUJFB-Rc^-Q^-Q#Fqr89Huc-!eS zQIi7emS>){U?}4u`<58Rf41`S-!lw=N+KnDKZErytNyt56Q<6uygtvHFY@7GfkrIINIfrnjOo|I%u<|l>C9Gs2+w12YS(M%%0~HA6Da~c zg%&X6d{y8^OrjRK)zeK?^lEYbpz!R=NIJ#)R@*zYmQi1i#v?1nrpU%ip8Dn_M~z%3m%*d=^l zA@K14n&*O{KgllA^wbw#8qq;;D*WU`=yXg$~O=@{;kw-f$*^9uFT{; zPi+$EtLuAWS7=(@?o+1or~Q*Cu)Mo=b67>aJ?1C}iYr)t(x?F0pJU+_Sf->u(utj( z?(E_3t*xHY#}bbOCH4G@;4tokj`{-VeU9Nq!xoX^CrXZ;GeS7MO;*2{oe<>{sOjo@ z96CWwu;gKqu@{<;n7AO!$(QiSCSfY=E|n9eC;D?a^qU1F^F|%#ZljdTjM=ih0jC?- zh^Q}H)o&I~c!Kw#>4(-#D!s2n;m*|Hi-q2c__}(MwEI>r{2a7xr0G8e9nmGT{iB>b z0rzdH11S&>{S!_>P+>Qu>Y!4HXkasJe(oK9z4PvV_{rsVmimcu>1|1nsX+}-gbw>D zdHnoJ)R@54K_QOgr8hsqo$Ggl{HBsOw^@dZX3pR}g5cJg)zB3GQ%`wwn;^Q=kJvoN zSOxdO*7)$BOAVps?`)GCwlkqTC3lDIZ5AjlI=;hBI1o=LK`RMxtbP4x$`&>$mKP826pB-(-SR;>DgDN6c;7WEBsc zu4YO(4!ibIRrpC!>QnLZGF*W$8F#w(npxU%?zNM@KEEh0t14kH%X~n)*vq;q!#_3N;t4=pu$dErylaGL3x-^(#`Nj;(`+&?Ospv=*p>k z9+!Ryw5W=4VVA3Ui)*W30jFM8p2p5kA*&p^3i)w*E`qPTV;o$ zN!vnmZ7yf8Eevu*0A*g=%@vO$wvCq>ShsL$K!(>HeQjdN!LNqV+IPJf^YW)Ql)4Iq zvSXNv_b9T^g!IsIOZ_pVMP*JTA^czyxqMpS{d~hIxSt#1{PEL@LFZk&S>{_DPrX+u zdOOTPd=wk|{Mtkre)ltZ-HJoBg^{^`yJL;7GTdi9q-Hk>?@~`l9pBi^l}|V*B09LB&ssd6*;98l`g9=pJ<*D04N7h~v&Y^fspC<~S=|*#97y1u!d1p!I39 z>W}u^s|gW2l4B2b@;&zPTHc4vDL#8Upf;VR^%IZOXh0!-E`RQ^GykqR3lt~4UkO5I z)T_b=3jJw?P1s%h%s^=RW%8qge>T^&I`VH~+o1cW6@rL)Rev~7w{ACNhtu@=BPG~W z*hQeI*8yGE&TL?lnX7D5z4LC;=6XhLO%OF$usy8RQKOW9VKBz%pUKtD9uxHN4cQL3 zq1O=QTT8`gmh_rFXV+)c>_{=9r;HDu|{tRRYT$#jNEJ&QEJMGo*#4AqYAF(T| z&c_-zeVfcOHjWmGWiELHh!AYzbmqn5Z0RjprNC0UxgbF4zYj?AfYFHx%CmmB_FYcc zN2BF_YS?8a1)btQ!jaaFUpKg7c$VIH>)rCuT(%Op*1DYfW&Dmve>B-{=rrhXNUAwz zE@=))E4(U@RZ+KOw;apq@xe}dGlkLGeiUtRos)W9q1zh=Te;;WIIcNnF#baOG}nW! z$fA!2MK>y!dTzYvHs%D(U&<7(C?{_| z9yllwSg6wUcmFMQ&{hGmbUr^7NB*1o=ub&1V5o-bKmS6A@&{oG;G01E7hvy_BsUnh zx&p>>cZ1o2YJku-VHSR*tbF zRl^*rUloeh)D0WCuefLiiEdPr+$hfq99#(;EpEhC)e0xlgsE1Tl>sn8bsF^%T#EN% zc=U*GV=)uev_aqT@gK??elt}j=Vc$RxJn==p-*kP^emzEQXq$_V`&N5*j2+>#YR?D z)QQnVIq`+^K!p07vO(O`8)ta^oJfzEvOP6rg!##rvw870zyvXNy=o5F!czj5uGW5) z;r#Tsi$%3B?i>T>M|^$-Su`-m(nb%ItHdn}yjcGJKsi z<->jFWM9j0;Is*wYc6##!`E&sGWa0JCOB@`nl#y`i|W-O4NR7atP59530qp$PUz`B z_Ol||zyBDwRM=aCD`+f!?bhR%LlrP4^w#jodz`MY;I)U-Ffl zN=NHBl-1GM1pz@z4PKfv`{Cofa)yJ#9WUd1hV8vqLI8ECZ75n$+8mes=8OXr8uGi7 zLT)A@^rg&r7LRE){`m0xs5MBL-_?K8jR;UU$SDvZVA1CVQe9p>207vz{1wCYgs`V(s8! zh8k-l&X}^Lo8X*4>|FtQmZ{|?;Y)Z>wQeaY8D6<_2+%A*XQ%qtiN204YE+v$f9O>5 zKo*z65&i=1mQuj2W>Ms4-WNE4y8LjS;nLw*pft48IBqdJV>c&lPCVx`&`m38_KbBD z&YaDjvhn%Y)5S-bm(o?*J5^tL5rLO zuTmF-VrpPY6gaW>Gp)4R!nBplRw1phZ%N@Z2Id0+5N&XPv?_Y(M8+3gp)ZIm!qu=NBg1j3h41DKRL}iGqGqWQ zL~|_6CHI<~V+5`6rIeiK7gUqxi49-QXIsyo5I~2<9}tc%5oq2oEU3tcbc>G&qN`4> z%rbxvn6!h^47@+!*o+=m_E+o{YSLBd&@pOK2&v%NMg1$KRbuXj8k(p4FW-woO zV1rmcAlh7K^jz07Hybw-1HR5RNhv{eo`SD2eUFRR?u1AmiVq2F(uimtl z<2Z9T-~$(YtN4)yvb?60k79gRe}@Q>8NX3k@$y@{V$>*pp7A5f9e6TGr;-J>k|}X zHi+7A(~|8d8*Qqu+Vt6QAMLNoUDkQrc`5zR!7RTCxxj)Uh|%08=7^2X^AwnHU&;*C zm)}37S?q1f48%I6R3cw+dUkUnx~B}QGPPL1)#JB-+6fYV(5o}R@=cp#Z+v+z18qWz zrMtk&qjvYvvW@Bh{kuH@SG)hW?*@I8Xt{CTLHnTm5b-ayw|qt;-H-$*r`i@P##rCdlt&y8jkGN z7*pjS_%dIAzenCU_Vs>Yf-Qnm%h7~54HSBdJRcWE#7>uUsk!@B)?am;v*o=)?}}|W z%vLZ!`Qa=Y<|bpylf%2N-?v15J)+4@d>@WL*vmaDz;rlgkNAyY)L45B>x;gZ!}XBi z(qQWHN)DIC#+I245EJw}|JmgPDlEtCgKG3b^TsOMkIRZoWYv|(u&qEvi}9Q2S2Pu+@m?pIHJUD}ou5Eh)?_E>0#BYIskYYn(My_(( zDoqo7+*tEBhx?IaY+qLNvxK~)R6+PIFdbC9k=0VE7EWx)o_*j7V$6izEKlH8tU!j< zwyFlQ)}FTzB|zm7CWcFc1?B@I>X!g?=*KQ18^t`eklj9e-wn2A&qOE0S5P-QQUMdf`vHc{L4GV2Kr^w5Zitca&^cPxf&w5Gg_+^6& zI8yu~Sp+GaQ(Ynrv)g)nhIUSxMeezdSHU8P+l5l z7+#9EfzBT`8gans1fW0nkH-fiu3u#)nco{yfekyqLw@NF(AEdJZ)AJ_12QW{>_kKe za;CY{*V)e5PhK{(ps+B%V5i_R5*W+B2{Ds6ja^C{t@)1UDi=OZ>$3Em$s%6*jdKRW zuGb5*u3h%`grhbm*Xxl@Vn#@-?(5?j&k{CVyFxOyh*=m(F*?)rmXu$`Z$sWNPpYVc zXPazCZ8<15TK;>DIaEdZuSAhQ0mGl!C=q`b$bWG8-;Dp@WB>0aj{nvRA2i!n-%(F` zoTau@Ia_M>5vUinXZdEgTI9}zqxG&*5v>QnLZ|5Va<#Hmb;L}O3B%8QAGHGB9C<>x zf;b{y;s=bcBPlHIa(Zh69Svnh+06clK=}Bm6XVZXb3e#Xed1VeI&ej={uOpW@^Rfx)kUaIUk1j!^NqnKcX1}f2s)cpT%(C}38eX_ ziQ|J_Bb9xcTztN6=)6>Yeb}jy>p+PaL%HfW--*#(a`~v*GizB-Z>hu!350<~0aX+I z;Fjv;aL&L>_jvC|hO+~^eUIxPZfrERTY#-GUG}aqA9UzXbaU`5_j8V|g{b7NG|Fd9 zUG>Yne(M+eqnANeWKRAR)D@`90nqe-^l>50&9T$G&tYdMqx`V!VArx-XVYO|L8!aC zNgZ&g7OSAitYK3hU$Mw^M?JqHaBYQb{o@{;jUdP%mt*_lG8?`n_adAZ(l7%~u#0z44dT{%$e*%OvC|M{*z3FE z$abo4AEf;NTN3aWElSC^DLxhIJ3}rI=lToY1#j+9QP3OU!cY(VtJMef$v-Sx9v~nd zQ9h7TXo(C#K1F5OS*4|L3B1^#v;}2_c%~hIh!x1klc3mIkjs0Dqtv!$URY5Qsy;3i z-(&k&$Jy=;SsDq$<_8$1Q0YFz=hfxu8u0H4i(APxHoJoHo}q+9-AOoSYnh_VVr4Ie zO_g-L+z&M6LjV}MRU7#ynShDhX<-vNp^w*V+*^{pA2lG#mfqY*FOoT~)4HXsYW&7Y zm8U(U*e1|1h5Z7c{wl;5MAWoVj;tD(Nc`}b=i#$qNh+Eb@zy4#E zp%&O@Z>;)mu9axOJSlu>-B#d3A1r|Gf6~06vUHf`UKCH+VP7t~7W5Z|1L4$VCm~qZ%jE(T@ZTW3l4`%(l#@`bcLdb?S*fF53st)w1%8 z2U#vIu9aeBi9#Eya9)Z4hO3H?9MU~o2X@`7&aorApN*BJ3Q_%CwvNT)M4xUAwe`SX zGS*5b%Y1oSa+jj;IoddX9y=F{8A5v8N^5|xC8l?`Cb$LpAxFxde12*1^3KxpXc^Ub zaX9zx?%Q^jN#|R}4kV6992pK(t8Rc|)fTcfLM3cF{X)FLMc-c^!rs`t6?3%XQ8`i* zoHojcL`d%WLIo(fm=t-6r4zGNHZu6EFv3s$u~-x*``tF^Zg;a@pNwEE|24 zcqjVFu`-jEU#|xhTwktO;@Km20pHxP98~Tyj_@R@ypnyc!@)wSO&?oWyI5y`z@KRt zZ#dWUVBC8d6Micr6kVdVWmpw7m25DNlNnSeFSe+MYL{>tJIs6FidCC?!t1<}cYL&$ ziQ0%d3?#{#13hBK{BggEZ+GzBOGy`6w>oZKedD2f9HY4eB$id}JXV4ifeV>$2e%Vf zQd}nUF1ocC74mLp%h}d!rX|rLz&Lk%WQ3 z;)hSy zDK^Y^GsNQAjvP9PJ+8$Qx35|{&s0z^`GeO2ddSQ-gn+mWu z+g~t$5!cMmK~L=2Fyo4eg;#WJm95Oz@bnaR3%$3qEUf<=qUf&X>rfqbeOn16o72$P z0A>SR@M$+vNXAR34r|PiEa|I`wfM9M<+n$e`irW|L;CbXWCLAk%MdW6J%MO^F(`9k=8L&j z6RbSt-Wg0Og2nfcCgYY8CfdxcwcC!&<*6GbtrqGE_m-8%w=zg4q@IdAm96XSp3;ep zFaL7W;5YkN^3HrKdf5zNC$V!?c7ronuQ=Lxl9!&0EFsW73Ag zII`Y9O6{GEsp}_Jtl!5~wS=HKosFtcB#s^|JV%Ck}(Tg%3}WhRNx1*YG3_5zx*y^ z5gjK(Q-e%SlaNv}F6sR&vxVf>lijf%2*IQDB;ykDsvXX$7@>lC{dY@Qr3{J^s-DJ@D-#=7(3^wxgL4SF+XQ-#`?E>!GB8Ifa6P)OPuMM3Zo(>Im| zX3RL3<&%20*&f1K(CBo8^SC?)I*CJ&&+K-if_iiNL;x?!gYO!WPv*PL)T}V`x97{QxW!>FbHGf3#IB_ER)~XlN)|%>&Fo<3$ zqTK#Iu06ykxuXPUlPY|=Jf7p#vA%84YU~$Sd=#N&mWDHUe+4+ul6U>}&^M?HJpnN7 zdD{sLj+pEfRTlh(Z@(2nmZ3j3Kq^20oSE2VU{K3ZQ*OK4EHVK%n{1YE6STW$CsBZ*4iB&{R?Dv`9PSu3kUy$f$G7B;C~CJQWuYjdVFI( z9f+wktC1YdtXksE0(oS*&^MHtQ1?MfUSr?4o20)MebR)n$}&7*p8c=z)Nk3Kh9P#> zxOU|oqxEwAt}pQ;<4yvZBC~yt*f?81@LU-0(IUC6^rpKEG@iz>7o11pDL+bRy({O= zXZhvWF~U{OFA5jVap%2`7Ovl6tDl!9H_nbiL32MLQA_x4;U>I8Lo{Ao1@`O!iqu#c`( zty*|Ag(?g)?zveqkvWPXf0t(+z0Lfj@h8Ihi~%5=b!g%~P*`2Insa9>-xSfRj9d)z z*1;5Lp`S_b{~S5|tq-5av(Tt}o?I_ZS7!xUkhedfwtErlW{hNfx;TPV*956=eB?JO7c+No{SGQZO zzgf4c%7+5z)dAmim{1^+E$)LtZ8pRrG}c&Rlig%CSL}#67@z)w*H&jMx74WAO-(2g z0(gh?1nl>8WE$*hnxpfo?)ufP{D)B;ePa`%3gCSALJTN$|Iw3)?(DguM2 zyQuToRd_Uc#Gi5JB^fD0c~73oavP}`qu3J%d89FLcWC><52E_0{meR!so^QGVbB%@ z0OL1BxsdmW`oe|xWNU@j8>QBoNt62`${CBVTaS-?I{YjG>@|xW8s7Sh6LVO5^D;5V zdOx=$j%}1zRKb0}?Wx%M<`0zGu4MZaALydTQMX87+dSU~ynO2jCv*q(q*J=E=^w)w zg%_QMv!32~TPHm*=s+6W7~hZto=C$%@57BK--jk%rb>hT^N;T;k9v|NkN3XtLbYuD zmBZ=>EwD+&H3wZ#)PrayG;sj?d(eN0_bv!|hkl2gv-Zq%`ZT_vny1ptfS?HI?=(=T z5e-c#u~$<)Kd>9(lTod_i((+_xaHO9c0GBhGaX={s>pqTB?c?rS3o= z*QzIFH#JJp|ET@{`FG1jpdBB2Y(-o?-qsuy?haKPCEWW)FGY$* zjpx?7kB@`9eo1cdaG!s|&nTrA^w>9MC9Nw^Zm4C2dkyYtvi-u3W@Fv)0(ci8zhG=~ zfa=Gl1~&|81BF9cVZnko>X}lSg-=75nvzG?O9y;Y#XXMvY$K%k&Z9H!OyZ+uhC)oa z-h&7;A^nW5G&*}7xAEMgHesMI&BKUVPFJj`YlqLMG92a>x)JMMM<;2#}>J=2M22wIzecK#W9`4VOPo7$PaRIK~2I* zM(!Q4fz{nAN@G@JlYh2Jm(-Ps~!73N~#KlB#~S zqy^-V5~p`<|8w;1imc`@A-e3m%FFo0u~3myrIctd_t9uEk#2j|tHK}eJ!(0iQV!o* zyGT?7zy(+0S5RN$n}khe+-PzLO6Vyflq5Nhh$LdS@a;+8vjGnzb#WB#u#Q?$*W#~1k z!#|ihD)7SlaFZK_HzAhB9E~Qt9L|C7rcauP09X%-2 zsXDxmI8K1$vn))aqZl49H0Nj2syL(6T=}E)det@A)RSAA31}S>LvOhdK)NOuslftoRa{k_U^Jz#APRM zt!NN!HhKU$|9cBf9BlPi%m7^-?VeA}&QT%dK6l!2q#4-`$E-Hvd^V@f<&WR_oj>~he*0&SZIA8q-S_kO zem>vN=k7p_hMK5CISQ4&;@}48Jmk<^)G6Oe7rxOvFUIBYB!?1PJ?Y%n zCHg*K@G;2u1|-$EmwoZ7p!C|Yb-gaU4J1Pw7b8S&e&2>{TzLas%gS$cT6U@OX-Pl_ zaMM-Pb^Rt0N*Zd`rA9anDSH!|E5_SXy$`?n+E3LzVH&miGqEaLsD-}c12<`TO>3ve z#nCgQkvua;H zO}`kcTJxJg`2jrn!?z5D!!7AyS+-J}f$@8z$$&ypnK9q%23g#6Z{x03l2Lzt@zDdV z5A$tx7(RXbK-N5b2*Qb+leG+S^F~iS&q24S>4}?iE!HrP9Hrq175hRSlICK#yWIc4 zcKTJ>AQ|7Zjdqe_@1y6zdW1fvir7{OYx$jdeC9}kLQgxV?+Lm^!qoZ{QfD<#S%Je} zyMOj{4jiRu6DSrsvj}=|bBwLHX_N=Hd2>(HJz~hWM(|W|O&>)KomW~HdSG<-y=dg) z)TaF6+N?P_oRO$~-PMA$0+`)?9_Q3&sUh#ttTI6>=7sQ7To^-n&`Y-b4;Knj>PImV z^(v5WW4gfMMZ|5|@xVoQI=DvU2a>yd5W>=vEPudNk!s@qjNO8V7RY&iUiK*1ctjII zt>x69mD^6DGqUMg1dWuCH?aV5zTbxgSu)ge(pel)+8}TIj+23l)t5tTR-`6vGifk5 z5RyEP=Zk^OcNofQ3gA(RCp1Rgz`zz|>4Elbvrs*UV=>It7Jk?7=W<^SamH8{BaOY6 zby4m2j$BkQf29Wl5s`gS=a!3TdNM&an@YE0a3Q{CNfutN-sUAN?$XpLXDM9N0Gs=0 zWiI~z%VNak9@Yh5JiB4EjC%t?ksXRM@4fZa`okCc{Lf9cY~-|S6CQTsCcn6|qTg0n5h>UKGz3)FkE8V6xBC#GpdgIU?h*|C zTHy9==dl_;bj^No(=vbs7xKsf8?JQ=aH;+=0O8hpc(>qA?*9yd`tNbA5UPn*s?33w zPhgJrN21N5sXWxrJrbLuJ#Ce!R3aOP{^9aOb}$85lP6w|nerRPn@_x(Ty6NifcRal zKGjoJFXZ8A&g1S0^HF}q?ZV2Q02dPL^9ZN#@qX)NHd$J_!P>05AYv^VW*Laa@LTX> zaXu(;AFK1<^%ky&u}1@EBV`}W*;v^6)TNwD6p%{}X|XfZ){MR~U6gNI*(`mdzazsF1qB$U6C06$cHh&V|8 zV{HZeK6v~|Zu!V>ow9nvqOAGPu+#VporEnvjpYoFN+$m`#1b&pF!>Ip@chYb?W< z#OUkZgY4bicdf6>YO}hj)un50SR#BTb_O%QgJCgmj1=3+=a%D>i&?X3m_uaRb@jQk zxS3;S63-5fSW!FX8`+I#1f8U&0hWW^P7N;k=v$wVGP=xnq(WXAN_QfXLKMr}pBxO- zmJv<3?ZESGu!b#Mg)E#=yoI_#v3O&(n{Udq@Q@nf)}Njzp16Fvs=s$0pI=~UHXd;7 z_(|f3JT3^z>3n-;68^?lZEcjlTvzvs&lp0z8-1vmhddtUeXQV6W63am>0JO(62&ce zexaFaRmhWO9OsTo@jJX#m$9Z_?!yj@3Di?TH{2lmUAu+q>PJRKI9kBHXKvdW*pct_ z6#FlaE{coUZ3vczUbw*b4?8Kw<9Py42+fsW3=mW zQr99U;Q1TOjNUI_!cs!tE-o4~itb?N<7xuZGqLpYxZD_R_Ndw(GofkG+#WPe3Q4@_ z5GD(Ww_@l!-c89Ew3{V5ov7hn;H4yE;WPJg9L$D>pDicOI)z`WM2N;VGi)qbmxhcP z1b;V5@omG`)z+%lGsQA1VBs%hy32ztHOs+4WJ~DBj8D@m)vc~_PnfrH3_|-;CbO9C zC!JazEq8F-H{n&d8cXceoO=0_Gu`Ou=uN^t1DhN#ZpcYh96h@Qo6)6zVhawG*&o+) zZ=cnxCr)RCTaI|0&?)VO4}61+*PL)244ZCRiH=+GbyJ}x#kC(?(CECeZFYW?N%?x3 z)lDjAm8lz)>0K>J<1cwpmalXd+BJJ&`5n#@rlF1Gg2hE;L8Im};4BbNO8rD=lMeHF zP)^K94&NWh%&oP*DebKIu5S8S6x4u}F)!?az2vXYcV(lj_I#f_B-bNqqhw2Mec2zn z8=fk?YXioq5sso+>Bq-6c|y*c_&j-emoT!x4K97_5W{|?^WtqL>(f{zq}#csppu_0 z%eG zG1)VV8eLA%sn;v#<&lj(`zxVi{7JmV5q?dHSQZHEXt~~FJ4RUYXePFfd8po~=Z&6R zTwTE+H^|dmEy6mL=jcIg+VK)xY+9pEUu;jIzUA?WV^M_bOM1A_q}V_&4mS=3!*>)_ z#!v)j$5p^;bCJ@m^+Tvae~eRX%EpU!+KdOUn$+IWC%t;qAx?3Y2}%m1aSG%%EcrE9 z4vlAzr<5JzkW3^x3n@KA3My>Rv0k5~)sZhYWO4ar2e6(C9%KW41wK*5)BE1A>pj*g z!ad@S3KZgK(`h`K^`Mx!w1b4r>X`cFn>T#7g^~N13wf;`s;&1a^rbp~Z?igHMOGVq z(6U%?e47R9!(Gy!Medv~x+pANxCJwdC;&m3(k5vLM=1!Y34hX4Qo diff --git a/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png b/static/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png new file mode 100644 index 0000000000000000000000000000000000000000..eb7c99fd655ab5e0b5691f41c066a080eb7b447c GIT binary patch literal 50232 zcmZ^LXIK+m*KQOMX@Y`)A_N{3u~0>jE}$a4Nv{!6kc1*lkN{Fuq$o<2-g{`F6AU6s zm6|~4NDI9rl#t|1eBSRp-;ZZK zcG}O0PPf9RAW(a+=6%)2uWZ-IQK?r=;QQo#uxxPV2TeA*Ga?#R9UpOpLQc=5-j+Gt z|HkEXZ*HaX+=1kxYhKPWTQubBw`We$Z+}RByUX#Mle2eyeMPg!|3zBy=&V)2Xm@k$hixe)0Z<8zq+$=n8NI_XcZE^Ur5 zQ;&@2*KShHN^@4cThpf=7XRmB7Dqf98Q)KhIyz|CFOr=;FeMK zWD2TV{^)s&PQ{S7mrYCblnj*l->35oJ`RT!*d0YTu5LJbkE~8-+7gP*ilW3CAGHu* zvPI&GM3ip*;_6BUL8?U$GvG~-@wb|Mh!y_#c8SkLOt{vaKXt-^AENd* z?qrP@P&32}gUGRoEaYn2e9WQ0KW|H!tNIIG+Kq7vyE|^fx2IpL!sm-PA}AK(@%WRV z?}+uU{s+OVg)b%#t*dl{3iz9T)p*Z4jPN`tKY4K!rKuEPS$=JoWNoDXZuJCU!WQ_*#(!)9AHpp5_Q)&i>;%`;)8&kI z@5b_}hi)afxbf3c1!g1B-~9gE35R+7UL-J1oB^gC2%#|XdYbH5Pn;=GHWtIEkQN&C z>RCaV?GCx|VNYyy1~+a79aQ_{6?h$I5~gN_2fPD*=huzE9>M663li7-hUKh@s>y7a z;+aO0pC0Y~`@9b@LwVSnq@AyGK1ZH++`j1C-jZI^Vtf3&awTA}jVk1dGrnN8;NUxD za{7fXBsG6E0(Y_vtILCA^hM>NB;W>y?}L6ht|t&ugD$3%dN*~Pd{#*+nTz{2gFN*s z-ihaI zKbJ3kGy)asa3X)jc4>aIC_W0mP@77Bws&vRnYx$ud}Mk8S!he*4mwu@9$NL{vq@AK z7vNBSoWd5Q*52U3>nCMFpda~S zPm7g}!!W1Tg;|>-q(mWBgk*Qw0Lq|UKpZ6+S)qK``<4^A0y`Ky1wx$m`%u$tz8{P8 zesx?9-S5kj5?S8uUa$+?xlQoRurkD0`?gFb-ut^`idZu#?KyU~@VhtFD`H~5jE?l9 z%a-kYa4Hjyn{@)vs3~>FWl?&UKnvx*Exm>^D))0>KD?F=ID1Y7{8-TNC|T(EgUMiq zO_fgwT$mc1WW#%fnwepj@txeTUAZJ50rRlid?la3up(ByYP{A6-V-Q*za zakt8ZU+reyPUz$2=;JBfaed{mjvZxSRE`2dNnnIIJjV4DTyi zjUC;StH4o6VK6}(J4@H3{+CKt1*lw&aeBEggSl$sV2|#^DH+q9F^G><7 z>TP}NUf!D>*sQ=sii7SwU@#vtI)nSIJGqqYo<{C5fakM+#+-tqkz3V|8q+F-JiL#W z(wF`3V}t%_8kOAf+sZ)6hr_%v_ZObuE-=HI$|iVG8hf7KUXe5^)5iXA$OYN!x<9vS z7gyMi%A+K@5mXLJCmY)YwqiRo3}yLWS@~$BQbz~c>stHcTJs*e<@|L{4y8M6#_0k4 z%->8d`7?IyDl)cJ&D>hmCy~l1EkaP-JoEV%Dk3LAENGdY^rQX=*EU+OG2`~C5NGY;K%g(b zD9^0+Dx{8r{-CMDLqQGKrODcH!sPSJeB>Ro;k-l}u3Eou^xT}?VrF%%R64wseung3 zzkjkV_vi@jUn37PeA2oY!RUP~x27`urGlRLl}KWmW?qRbLGFLr3Upy9E`h(t$y-b_ zj)O8UKzN_U9?JM6_pyVqmHZ}i{4b>NXYnD z+;1MJm%Ex;Y&#NJMYi9Vv)8pXB?@B2+)BCwkfgap{@9S3J?8V$Ew@T@>NHnDZRWKR zu%=%s{B~w;{=3l&iT^A&VY~{7v_PXoNMj{*fho{sT7d1Fp3Mi`F>MfUl7--HoXQ3# zLBh8a5TYc8WPa{74#})vT$>IhI-b+fH|~AXJfSB>)#$L-6K29OEj&-UKc)u{O6ecV zmo$TSO*4F1M3iOOsbRUq`cKwN8T6QVD(hhL;cIPvpenebK<*VJ=I7yG>Q;Ky;>gWU?%7YPt2x!8+7m)oR7hD|5?8{&Z*?95sL8Gcku~W6zF3iujA_QfN({piJQW%5C`zw-e$Z?lc%w_U(i&%Uj}^ecQ=R?uQM zJ1CPM@F$?QQ)tBc0nGEo26BGmi|Alg+KrZWq?oT$GN?zTSXi;~*=QZTzE9H)#ohJC zxG{}gw8q>gSip?QX~;-Un4|eJ{s41m^V&7lKACN-xXp0G#d!JzsO|1KB!G5wGJ#ufkuiZUHxlvbB=BQ5|L$mf4j5%cp6R$fSC@Gb z6zq|-jwoA{)D&m_Hjb7_u8&9ci>~q#}?yJtvbS)|tRPj1H+QjbZdY!Wn2g zrjBNfRI9b5ecto%Q!kt!Yt&vHl>c`0Qrbt9m%4Jt+iCW#Jbv?rxzZfc;Q?-d0Qi02v_c^XBTrxt=UC=Ew*iKuJ} zMq#Q*to2pJp@7en-%=5#l~{|P5#79_`n5(hKa%xTad7V`w^Wu30ZyC@r^PtOM8ADx zY(#^4h!tY&dvBh;J~OADB3cXP_ENN-Pd~h*MJ-R}5e|o#4oSUZD=JmsFJbHtkvEJRRe+Y*YOG)N&8v-s z`6(FDCgTA+a^#>Rh@uK#{jTXvL!gu+H&+wii9-)UU+AWcZ|w@3vAjFZ_t#atkgwP~ zoP*_~6W@i!i8gPqx=z=)rfOzAfw5=NCAVwH6O>rR9eO>j>Vi&w5>92oi@YBlTHU+22?1^0{_Q zS4*YrnhlR*P(_qQ_@)_Y->$gapUgXKcf4R%to=TNice$Eo$={@%~s@6(PRt%;{Z0B zCeNbYuiN`$nYZx&$idS-jYfWZv>$iB`kSix^yv#GBw{VQ`F%KI-3w)~|K`DYo~Puq zp-ITNoEi0s^3A5)){ACb#SMs@B3ChIV{-q}tu!<6r7ySx`zy;iMe!T5 z=59EhA3TH{$}5>Z?;@AUE%Vdegrq7tHwAIL7dU8ytxmTg)?IO>F+}e4Hc9+*E47C* zNC}(w?yN~sX`!*JmR;MRw35#6y@L9bT&mpONI1X2(K^=e6|i1qH0FaQT**tlqc2FF3h*poGUc6xusrTCb*EeSrf)yd5lI`oAN%+o#%QBNu4%p z+f--54L0hL0_Wu|3O^B1f=_ok#WP3&XS2y~=VNg-4mvT$F&qdmG77CmSsd*3(Ec&C#`<3f_i=8F5gd2qXx`{&N{cw#D)bSscVR1@7xC1Gn8y4lsE%B85+neQ+N)2~i z&(b}aqxVy-J4df<_*!>X3lp`cC@qDS5Mb7Ofnd%-(?sGd=YZqxU*I$R#KU5mn^g5ZvElzUNLq(&o8z?S!Ue@z1`XUX*y4>#98kc*-MgM|3bPc&DXxH=)EiU4>XnQZ+wZ{W5b0x5_z+arAXd*$N^WR zEd;w5g5>wj7|{X?>DtNOC9O^uQCo;%{!8CdM;0nxr!nG)cK*rCOrm`CEfiL>lGy3j z8S;n6Yz;Xu)8R0>GKdBut(@1kR2!luQYwy#rVB0=H$0FiN&yL;p{rsSe5@erJ^SG5 zCH?cCM*3~Ee}1aTS%7uGk^-eKUlUDzodq5Ul>e$<>u{?McF`r@g&g5vJhb_hc>*6p z4V#ZmT$sOJw>hETOQ?0a$uc={n`4LCV?c)Ycq)Xt2=hsT!zoGq} zRF6Zhp%QOqhi_we09MBS_hLhE@9C#}M>Yp_87;Z|)N$G2-pxZ7@3DnMUXpmr@I=RA zD3V#c`S&T<{scACpBN4}js}$AAIy)QOV-u_$wAFn5_>nlGgWQQ*f>*Alo-7md zYO>&shCFPqKPlYfbmo5c_7y=382bovGnY@)?WSu^S%&50L?rnwQD{$y)hngb9O^=0 z5AsAknT*liN0iAKxCFTA+@-uqmf)X6Js}U|!!a+WPUOyw`huf+Ok?~moLdc0FF__45| z6gP6Az2^5*;O%P8dvq3ie?^nz);&3M4>6FY6+gr24EMkths4oshewP5B==8kI;@HC z+3RvRv)p@R_t-#$lPO%S=sjgpgxm+cJ?oIGX<*9R75*)S4IRQ>+AbUM0k3m|q23e)6P`B&YqN1_>~@~13+Pb(hb%}|XKiDxJG_+S~wdrQN+1%cVN6cn`r zdp19<>7Fb%db`72h`Z;{?1Sdk7j@;Xjnsqke8sKGwQzrg&C55obv9Sl9US1-y_{y$ zJ{VZNijV%~!VDg~mlqc)Psq}&uW*MeNBZ#|28PUZIIWu9WAYzz7m1G*T&&aWuYMn3 ztFx-n3jk-gQd37?xH4Y$XVTaIQTsiOhR zMXjm2D)RolwV0y*QpHjVK_xKDcGj&1f`=C7ArY)i+PVlIXoJ%s!`>Tx zxT%fjq&FTO1%6E3`cAcjdl0(uGre_%w0xg&)|6kV7yzwi_%ABQ78Cr~k3g>+uTJQC~^l`SLIKWP(7T3+v`pm4-DIF-^VzpY7pn=VybiI{hX5Jlb9@KCV7^wrWPz1aQ9 z@bP?vQK4z6)g=!nJh=MnkD0rHUlTG}1)w-`N;^{S4)wQOUFi)n8ae->(&@QJ2^z@} zRhuR7w?pxt%NTLbCJI!j^t-ZSlsR##k&VyT!lE0pg`^ZzMxkt1q$rH%(y;$CJ4$TM ziWIfrLHc`2T8r*OYIo2r2BlS;sf%hG|# zQIB_XrZ1$}jy|T9c7N#ik&PxjsiAmXw+2J_-n?%KxcOBCN{KPEpd4a#Y+i`M^(vpA zzKqnwIXtfohbc=igRe!m%WxOcpGdaIIOSTaicP+72Ry07jZxW;Yi$y1_7q)V^FeNXxfAZ-v!ld}5^a4NM|(>>_6O%*w_9Kk`DyDN7Rpu5j2B?1j_MfXe5fk2O@K3f2xk|8FCCVZL>> zlOy>w*z*BfK|N5xcT|g)h^uVp`geZw1!jqzi)7(_3FoxjCvESKvv7m~i?)N?VNq4iV^9V_J3zsprCGP@$SapFJ2 ziaR@=PWl!*!sjbmbhEr!YjW!cy*F--VT8eJ#%Bg5?5?ocygPh+tV|eaQWgZju-v{Q zgX$JxIE~m>{jMGPhnZbxQY4_7^rsJ*Hlzgg{5C$(oPzPqW+m_xJ?@ez39dl0|7%Y5 z0pA+$h7w(FAW!8<_C!(Aj{1D<4EB|`eDPf+tJwI7j2UOT9 zkEOh~7WWG)s6!r=e4w~7M32gpU+0j=DTO|(V8$}~R z=p*#b!@*~|O_7-FAaX@*>t#1HbIbnKG|TDb(WzTr~<&e29309*`g zg4#nGNfiBJ4&MHf+W=ax6a|3*zxlUsj?I7xlHRf!HJ%a|hIXr6<}SYfCe5S&N zF2aPpCtftKmW@zq5Wii{(*5USUD;gREKA5~y6o2~-Yp;O^`1Q?KiT)E414IBhfk0|uFj}0>`c7;8=d*Sh zNCTr&u7U1Z`x^wf=Dr~&(ZW%^_=)S;Rq7AVn|v*cZJtIMI!TUcZuGejq8`4 z_UFbDd+TEG;_APHa&)K1@;9BBf%n)eFYiR!;m|L>E$*qo3(Dk0%g~xKm;>lRou_~U z*4(gt8GG}6Y@TTkg=>$`*0LU42X7ENo47Y=8)=Hs!RBCaZPzA+r~4T02e!BgF@5N2 z>ihbXWm_l#zCv#D7(Jc*P7sYOC%w$=d=z;Wr>9I6$Co#4ngHxaTI1SI_sD8_y;{rS z9DcxXeJ*Oh80C*qB^+SVAFIGuFn=Q)KqUrq{!h@R&5o)ugy@-&E=SGak!w$-PQ*?l zl@Z;fxs#Rlb}J4gMDK}b{2_VkxhL%JAf+U_&>dc>C%jqOpaK_1tLKYjJdfltvI1-8 zk;#F&cg1Pcq-Ep2viBrGwh-`4X1gR}5fIQGs;0K)=rX`c><$|n=wg6k-|x=b*D~&K zUjPbfgDBp->0}rHC0}YvB%EpI;hp(O_^$9Jd!`3}9{&seqT!}COV(UlXPBz4?@KvD zrz^Fo$1S&QRCvw&pN|P_-T&sr2O}&O##{a}-7mBBV?~f1PnTB3(W*#KfNBj;Ml5@Pjrdso%`}ny7IXb& zj;fl|+GswA&Ean`tnYYYe(PbS=IqDB@JHq86RtdScf!9Yup=89x?&0}#3ta^dXh2H zb`Q^sXhE!eO3Cv`-SyE|Th$q6vdtc<@Z)qxLSE28apO`e!!Qlwiq+4Y5o|77Ju-eGuQG-J}W(aLn?o8Bn5B+q-gjDTq)fzNw(FEEp7BaY3X zCCqI-FXbju?m}Id&-)5Mhk{CUr#pzFbh0#OSunL-9!e`>Nyf&s0@33gj`n;Ik{iPN z&~+D*dO1M%{6_tZ)|vRK%v;Q9(0YzhN!%%KfCWcn_@p9iFXbO9pH}_E?v*U}QLVD~ z6B;RUTmnHdFHJ?~i0-|Fb?z>9a*JrJ|3F`FSI%Oj^^)t@>x8%@|G-%FE+>ydYSsaHoCXw9wxv~f2+3nrArU%?`;OjFG z92968XfcfT#Y61F?QjGP8T}9RwZG>2SoO7+okMCZf{Ns?{#usGNlU|g&eXbq44|zK zK)>wm0oQ4F2KNkC8VLi~e%c+Jk%lyK{WYa%bCgUYYbH z-?dA%S8I7{XIv}>WfpI(X7+%eOSetJeD-APW#-0<#hnXD3d|G!`f_$k;?0y#!U`iU zz&@d`(!!SC%=c@kXGc1lNgEetdP)tu_?~S9yT=6P2_CpiMqOEm0X5AZn7M(D~<4M;f$D9AoT%h>E4;~Xxwa#shp zhzwSh-QeI$6^U!qRI>JrL@?sA?Ykl@S5_s0t8uw+tcGl!@wFqJP|o&AH6J%_!(sf% zWnh5%zsCpk(fWc2pOqfnofbWnKoLdabwx{ya3f?O=felSp&jdoGkB*;YL(Kid5KFW z<9-DxMfvEGxF^ZHmQ)MFYFuBReBo;0VE=3>fBT$O=SCAxX!$0t59zsnNPr*`dI~N> zwH)5Brc~0CC3e%49;Oh5;NNSqGAttv5N_D~pbh^{^VN<9#_!hdOX|rS#F!M<*z!B{ zcYIh3?Z|wJdi8y=9uM}%Fp}{7*-sbNZ#wJW#H~rE*RLN&nx1A(?^B^3SJ~)`yWFrr z%iPv~W{Y2xWqlIrStJ}VvZRUeeDPBaYs%Mt&~oFlmCfGvWPTI-H5Hi^K4;BGrV`&Q zj?A5Ttlz0xYHxj6xrFS1eUELH*)$&K2Ugm7VA06&mL?fIj1+;~%T>R`@vvhyX-~Ed z8hqha6uMTr<~$l%z zBDScD7^+9*BSiVtrxGSd_xsD)j#fz=P>x!`QyM-kR-^L#3M$TRStN=3SmJjM{UoHC zzlHIN&+}CA!|gls-RPvube( zz)U&f?pE>gdRNo$2qxSo@*X4Z*dKgj=gf74!ZA=G%8$ti(1R0xxusC?7$YSRR+knc zRk6=P5^g*m5Y(>Q;uDKaaI|k%SPvel(}p z#Qg9~Zr5N!Co+mZL$iUYH15NL9EsI9o1fA-RQKNu6bCJd zaR^+?SB7eztbMdo*WYWy5iS5AC*)%p0sj}Fz|&ylbc=aNgCPa&xi`w9U0=<$@i zR^fup7yq%|QP*Oy&pjt>U&N?oss4FuuI1`iDOkhIqiT{HQ$a{ziVq$?HmCsggyCC1 zd!r?5K0H^T`6fP?&Blu%$7%p|{K)MTOpec9cHph8)^)6bSh;U3?rn3#l!+Z3JX=~D z{Y>fnK`lcne$d){db=V=p$%I2rPcK~By3b%$eqj>B958{lFX+W`L5Dae>R#7-8;3a zdb1q-18|3}vJ8ds`00!{4eov{vu>AaoM}>-9(nS#yWAVz;K}z0R-wexdh)+5Of}5N zLb-t!#&=C0p|ACxl9k8)8ihT;KC8PZ4dwXpSPT6?GM8yTaotM$NMw|8?8rH&y}^kU z_uZi83DRv%T{aCZ1NX4CeabNVE^q|k<-%_wu|8d}yvAtUD&VUhe2Xo^a4zowcOZI7 zeaSH1SWcZ}LVE=4Cdh%P$5_zvvHl>>j@V4-ed=Y{p4}5$1Bvd7{9&_c_XNt0vScYCI8a{PLtcHo? zNIF;N_+$1+!Vt?9;pm!?)-I#M;P`6RcGF-hw;Lb_v%@@7VkI-`>M2AnG_v^drnA|^ zp>=u`B(9m&@n&o^EA9uER&Q}iTNi^;v@iIH#NxT76_(K_CYh6Wpsy(!;?E!{lqX>Z zjkXGh{VMD@y5KZ5V5Hr21&WAYp*O)|7V?FR=&Rg)-vv#1=Dx4%o<}+xg5UehgKlkl zQW3pX-)s>TYK_s8`zg#zf8#<|` zLjaMin;PWw;4E8R5z9Gbp2MfU*jH1Znn=h>cBIKwX5i|j*WRVb-R3J$by;uU$b~FT zzXFVd@>}MUU!=b7qKN@k9`0sS4_5lFuINdjN;$&~VX9eX)wXpD>S970cH`h_JGj}? z;fh-V%$r=pB-aKyaeSyFF?njtG{!|!>GIT@lpebVcztZ4KK=D!m*hJ4;{}=4&3v{I zIpa^n_&BsQ^-NKM@AI_hLcorg|3$RR z`J#y8lZCeqc2>g>B9?a0RE9f#pu`BPL(12k367(wEOEa=&NJw&8WqR8B85M`gjtTi zN$rY*CLzkA-0jpdY9!92F7uqmu>!MJ{6Qfp6_!r%8LSZnVKo7~EEA0o{U1nt4hbO+ zpOQ>xSME5W#f~_20&0N~FJX=eBMu=%QQF;r94Z~oh!bnizmco8Zx;^pORIuq@i{kr zm=^nri$I8`Khx`!2^Q`B@nNd&?h?5s{3=^5OU{~&5!K-8qK zj^#Qfp2B9AjkwC({8fraEA%?=NzHG%mq8c0W{S@<++Ar^AU;@5t$~e$uh(d`3rofo zubLGC%CXw`{a=wbhkT*uas95lNE}**Ux*%oqm}d8&fr8^*^#|~K%*!9tZH0*IOJhx zu(s#7p6FPmM0&N9i9kWt$*y}hCVe|wa|IJVx><1}?}Y0rTtJ$Ta7PGH7T#S)a=Xk5 z^Z7m5xClTR5}g~MdNO!2f|fxLdwxCF{dscPQ&nCANDSy`!-y34`LP{w!DQCV#WHBYn!~P-O}or` zD@o-dy|c0tbNFiFW{`jt8|W`98Vt)1jkUNM?Q5zzKi!)v399UkfQ9e%RT~8?rI?<+ z8(x!EPZs#8`2`aDdtVGb)`cj8Ui(d?XSBSSYw;|D4k)%}e8|*G^eZVdj#HBtO_}$n z5~~x30YO`VF*bj7Ya!miN;Ayu8>qNqNb&tQ)-=>pKq$2c(e-BLduWJ+H|svsz5;a09Xq+ zB1})~4)Q@JLklVgRZ}PlhS!~jx&c+&+OvOKm8jo4TXoNCpEu=y;7?zGaS^2=P;`VY zc#@X10S8izcvn~zC<_DJ3+)`pwAg9#8`@zn@b82ElSL4WZl$*&aKGOO@9H_4DvE_phPzob9U)#&1l>=A z;kK2BV!R+4$ih)#I zVq-)14qz1xI!CLCsSZ51-1s!)#m8g_OWH3!LsC`rW-Ij*4(DG`R+l9G?G8QfDg~PG zw#2%G{y%icz&n$ty%DI>(88QNo4d7vk(jn!6Ko^0|SH(6hCME2e_ zVD3D>-vDbdy`3*{O6r6WdvN`|A|~>hOrVMfc$Xk2D_%Qan!IEWKCD0LbNu$9WVV*` zRCpM{2OMaTIB)n4jogzQTqEp#Rt2QSyZxw`lS2|8R$p6N9naUH;=`Tfp(ILdkkM!n zATC8htU!p#|CToXODwYky|qTnLnZY?opx6b0cDo>FL%UyA?MI~Ru zX|AuDt2pE@y6nXn!40cR0>iPjt~$;JAH`i#Rjg!RG(F~U_hJ+cmmBh2Af1a89XV3= zOx_pb*WULOP75vXya^CNi;{yh`mDutX}F8hV5suYvb;Z`W63>`qT_GvIOvdTnTUw) zd~{+>`hORdb6;bhiOs|tTxl+>V1A?M49Cf|&}AF(2pmur9bml1mmdG=wp{Gr6Q6AD zxDT%t|L}zc>D8Jm|sAS*^9n@IvgP6%+sc;F?NV38(gVJ#^GNYWAO!EfY)bfffJU+Z2XNKby{d2t+ysuW`?}l|%gtbsAxev$)d!*xzK7T0bwnkpoFS}E(iAk1e;;-O=q{jEm#z9gsd3y2Ufh*f0 zkF-$&szcpqdH>X!vmmJKb*Mt172z85eVm6Gf$KO%!FTRpbno?YD;Jps#r}@QG2*Pw z;DX=nG2_twiy^J%3AV!;F#0#qUMHq~48h^2%T5^XQV1^B!`hEbVoRp4Up;#)rrsItRR()l2{uty1w}3ZILw z>#5GG$_06jgNG^UfNaOb4BWI^-Umr7Pc3H%PW$cu9QT>hu4jqd<86-oDm07hybkaT zRV{x>4%DlP_z3ZDjdM?wQ3k%V_pUb-P7fk_QD^rvhWE4LH_cX)K4oURT8z1l=$TA2 zT>5qtA!)VZq50!u8C&cdN4mQO%lexQEX3K+#6n$+YeS1bc?AyXX?9u;)_IdX9M+h` zHuEiwzD{>ZV;hhWv-cWGs;q8%ypJj25ZCU<;Fa)+1f_DDN!o^&Oj8ZaM zdhrz4rz|YCFa`Fa)mdS@qOAJWKjpk+zwjEBl-2k#-v&he5}k1y1Lw!!oUhZbf!?c==SGurf;pF!R}Cz9%oEb zAKcJa+zvaW`aS|GL4R!s%tsqBmf>u~x}0~U9lsMx;%OF~SJKGMpX02M)C~aLfGai8 zGJIzpvmP6WvsL9fq7$iMs`2dh9i-1{Wr**q^N8LJ1)L0XWAS5OsBU> zTLU?_E);hfmt>Ybjxf_*Wdh24E3VSu8wv0;xOW@FQJY;+g6VT@sg2vgYB6&~n5shL%a#e@eLX5iKJqvD?$SN>_WU=H>Ue(h0j#GpsP< z_{MqU2tkhJj^u}Tn^2bnUI>mKdG*AX>zbKdo71j1PM&;6V4I6D{FgS_Tn9B&8Gcnk znMN3XeBYW=*8ML@4yZD3^PXy$fi|I$QlFW|McPGCrA!lOgIiye78~gyRu&a&zIPHz zMePd2;yk+%w1Ga5d}DB#KRoMqUl*os*d_Zz#df2T8w0LaIEmX0#YVdW8Rv>PcdV4 zHr#P?QP)0Enh>QgZcdC6n#^r`btlrf>K5OByE!kNoKDEUiiXei0GtKb zm@5Dl`vkfka9owfiL@|G60YS{@>xOlCtO|kJ%g)}{`tCi!wpGb#eEVOs7s77NnwKm zbPLViA|Ajbr(wv-=X%O6VmAOQ>*Y?l&VAA} z`2C)CYLZWaGFks#7Hp#tX`^648+QCZe;Y=u(i+MKwz~ig1l!P<9Eq~OW&y3q1Fk06 z`2qH{mpi^Pbr{&jIh?ML!GZBzzybTyJm>s@J0{DIt!tKq30uxBYjT9)A-|U1u695Q zsQC;+YP5tDh#Xp6`b0_;-LVe1MoNtUtvPkcJSsD#?)Z+;&HeW^x61@YeAB5xqe_gt9W@&5Xi5|Hhm+ z%D=lZ0OU@Mi|vPQnYM@}OmLlsioNPS*|6bg3T#AB6MRQ1HGz=yq?Q*nc!TV>AC9Y8r3S`>BV;VGS7qvwYFO0Nd=OR~0p_<>bi7-qw0z?Oc z^N6_GzIcfL>-`fNEVv0}wOIysFmZA_iEP&C4sYlWaW#gbF6Ru*b1ML^_M&g1@nM7N zZbr~x|1Dn%b*`W1+fwc__~sBhdQIqH&182hmkGCK^J3NpY9UuyxVXq=5&%w}blnbn zJytCp0aLd2yY-cWbg_yEevxzUN2K2NQcb==sp(7WVwq3ohQG~6w-a2_DT3N`4d9mgxL77#IO)}`_8;VmEs24@a8dL7BxUv4w!wJ(2dKd~F zC&C?MTUufjNW2wm6M`KqYzxR=(i%F|4VsC`OdluzwZQhK^FOkoDLcZ4ICtZkn{j4-AAd?QkMjme?hr7 zMZoAus`u-XgW|SUckKSOf^B!AWN<$)u|XdB>m=CKXfUvIyY21RjNtKvL16wbHTo5_ z$`o=~E(x)tfSb^^>&eNH3C01Mcz$*M%qyN8u=vt=d|h#o8*yGm$cGL01i8L-+gZ7_ zIDK6`kx~;Y$yAvOAXp*Mp}}3r9tc-{ z1Gy2FcX$c`Y}1r@N#S<7Y?*AQwUPKltUUESQa7+N+3s`4&JJDdVTbjwcAy`?k{@T? z-|!s|9J)(er1yG)T17r*Kt3^(ZasdB!S(;soACu78g1@^unQ`|Ir_3q~?_#ZT;!|6d*F#U1hfjI9tY*>`A!Hb(P2~Z$blR!R6dfVLyuzy2)-r zy&DG1bjWpzj1h#^@vDHng~9~}tAt<1aqZ`bLaRxiF`c;a{x5@vvz^0vD(em()NZxI zHwfLiy9MehI!y!H+XT~U<%Q0wxUV=vm3Ff4EAYGZYx7Jvx3B8XLHT+E35O?RBV4#a zR)E73dLCp^OF1uv`W|9eNKbkrWW9)ci8?-(QN;zO9;io63MU+AizeiOwQ^w!L8@xM zX}f+uhB1yk=WV&gEI(-9PA_B_XOFwS3XA}a5M}F|LyhG}o&8g+p*fj&{nRQcVG@iy z>Eyi3MlxCM4&?*r(BPs z&VRl2$n!+1T?U!8F3_>F9T~IxnEoN!+V@NyOLfITQr@@E;OTbC?Fz1d4beM94SBnl z3rjD`uo^sy?{+_lYAaLma7RIN1AQKiVTS6ovenL^;R58oVPXRdF1EjHZ=z_*ZNR!| zUcg2jg!K=DwLNB1{+@tJatRzb2~^FpZb5O2gFMJ+gUbi@nA;GmZdQ5_k_EymkvAA5 zS&~bA0!Tk3fD!v}11U@P=7)Lk%$qcz!mCqxK=cWrZH!a+_si?EV8%uB6AA!51kAIx za^MS-nl<%1B4ApJR)}j_YoY!73409Cx0A}|<7*E{fEYqeLF_FyVJnM{ zyXFfVnC9IvIZRzW$DdU{^gxGRSoK~89u2z3ik4{`I)R=QQnUXZ&m-?;Ue~K5l5i?WUZ+Hj{(74e+Z`PN z%xwc5^%nCX<<$`Olb@aed&$BUY6VN!=g!hflYl)pRI#U>90h*+4A^?k!Z!9QA%;|i zCTQy7M+Kp?X;2XI+0lhb-Oq|IvY~)V%yNPnc2I-<5+<;)**9JMuAY_p9-A zTl^-j9jab!GwJV3;p{9ab|Af=dr+ryS0x)Ryz3e|P`jfjSF?j~R#Da`ozk=7M2?)! zhe2PD9PI7~r!P6hWXrMqaG2u$Xj0KgB6{LyihzJ50@5Y4Z?3TSKKtx*?j84>@4nwV zzA;Yz(GjxNnrp7P=A6&-E6?(@``l@{Y)9erywzEI9-pUVbAl~>>|i?2P3BXx~F&b6Pf_k*^h!46Brdm*UONdaWS-WRO| zd3(Y~J{q6T+M_&T%5kipZZ1Fao)PCSb9oZ;$a{qwrx`eDVR1I!V+VK{=zeg-$!4HJ zi8ViY3hc{Tf=KofkzVJ!#`+bvidWuB4ao40g-dK#Mu>*Ylr&AeI6NhOr&W1py?pB# z$!glpCT&e#Y9y2V$;Ud-;)HTVEFrXwb8P5E&{hl~yy@Ox>vY{{F+|Qn_kcz^7<@Tw zR@V1a$RQe?(jMf$;a^s3}Kd&9lmG@}=6SG%x%~;Ya zC67#D&i)XWBa zDoqws-Ls{Q>y~75a!v;RbkgI5sw*4L1lZgaxyXbFxTu_!@d75pbY2-607g&I3~x`E z-RDaXGJWiSmY%w-`e!{Iso8BJbVI=A$+ehAK|Bg-=1~z9CMM=xZm7!#HF0d10-EC6 z4ZjvA4j(wRIz3fP_ei-nGQC}Fiym+6q5jHkuqh#jChT%z+^E)4t=clz$C!+X9E{^b zxh914>dqct#0*Dwi-`pH_(MJ$qaPJR4*h;hE=3a%^URzNL?F)-=w`z5thbr5KAM6U zxKP7mw|nM?9g`0-xT8!wJ6;_#f6wzsG$Odzmb2-ME#2WX?3{kT)!M68gBz^&P)qKK z9PP_t2y-QTSbW=AjOg}N%`aTCCq%b@3Wk+2POAbuv^9Q1>**k>kCvdaAsopkrPz7t zRg*2TXBK~XVtFnzH4D1=jkmDP1NmeYgnY~?yltOqTD7<14&1Q~>A>y*4r zj3or}s0@mW&%g8Snn+x0URsKNV~-5TAhd+_<6=hOwa%WN>pJ&;?y83YD5HZf`$cyaPQ?cR;gcJ}fjyN@wW@YS9lR zXZqY&zF5{t#6<-hXboY{I2)Ej5E>Vrx|)i$&B~7sd6ho!P-zBQ-{<%v~YQ8nG;}*cdfj#q(585 zRn;I8_To2%#QpFlS<--;4#I9Wu!e?7l~)S{io_k?c5bZ3OU5d4l#!JlWC^E#=AJ)i z1D!IH>A#4O{nh`uh@8S-&Zk%*S%Bb|%4Vejw|y6NZ+*{7a9)uni;3&=>gAH=p67%U zJ>s}J^{;>>2^^)sx+-5_wzJ^Z9cpc+Yvwzyyb|$vUiaY|Ru?cBm6H6n*X>5nZK!?C zoOg=GKts*{60{6IF(uC?cYm7p%bvr|M-v}&Z$B}t-D)H@RdC&TXl6w10MwVXE1xv{ z)Q13Z3s60ufMqDELxJET_p7*iqVODrqRKu}nc`rynL_qw9Ad0(-S04+CoV7~3z&jhJg zJxB9}@r2_sy-VI%z2ep7;&f?}l4%id)O(WwiWRU>}4joh1e;>m4rTH1lP$PzpU8?I^ny0qv(`5>`pZC zebW-LMR-Sh*Y&CiBGXtujutRJ40ud*&amzmjD~6TtwJgU8BG4Z_zShs6s4dm{jM$=KTrtUL`dnn;d3^sx)nog zAhy(bnr+hu}u~ly=!0%Zo zdM}_QBs}JdpAFsAxBC!$gB&3sI21|18nYZ!hLstXdLWM@qFy>gJL-+hO#XtUow|Su zyK?ezRad^H@vxhORRP>+ZNlNkY!8x9+RA48rS}-d_WFJYzT_Pix4z-ET>HWrbrbjX zNurycajL0{^UC*Z_k|A+d(7%cDc+m*8o9~XWqZg1MS$|AP+34}6%tp}M{w6^opEG$o`H*Rwql^0FGmY4;Q`0#TmlWzV-jDWFf#mN{3cGBFfQbUq! z<&t3MIYigeslZ|sYMUfv@AfouqT{2^gMUX_Y+)To-3vPaltu7BpG{V#q+@&7I?xXEL9n(m0n)v95U z_x7}J4lojKJ0y4N4;hzpwKKxaMS(L+%zR^zTZ1!j5cxDs@n?c3Rio7$*?UREckRFz z1IVXV79Pg*#BPt}0SgYUJAI~pk0CLU>i4M~MRs790$L(34viwfz?$K_zruDc4aR#T zw;|O5ZX^TaN0ch0>JC3Y-fPc9_|a}M<#@+R_C?Ld>H|o{4d;1o%<**Jm6XTjqBpwy zq6=mmFf|di3%1$2+qN|UN}e4n!soY2Jq?MYX@!-G z7bwCpaqLVh@gp*HDgu+dJ$~GoGA!kLsnn7BQ4`{-{vN0i`2Cy7LM;`{m<+-b zg4(GqnAQ$8jc?tyBx=a@%BUjwpDHa^nn$-{^#LoKis}pCpvY@T`KZhvXuJc>fc#q? zgCHwVa5NYRyz+Pppf#DQ|0KvQ6XR>J(cQ`5FWbnf&-Bw<8a+xm9h9J4FKF z`te%o_q2BE8IIkIycH0hP|=5>!^bJ65mX|w5Hu&!0DC43{sS)X3mS#L@lY0n(QO81 zx^#4OcD5$^;?Eh%MQt7$wS>!+S%Mtu9eny^qO-Rer=bK`>Q3|`pOYcp$m>LHm%;7h zZf%0)ldla26*Gl^3*iet5d)&Z9u;y+RqUnauvK z_+uMqjfq?SiBQdX)!ha|!4sgVxc+(~Rpza$k4ueNuC}GcA8@r>d19!qAF)s5sOfPR z-nKs8xI4lFeJ^wA2&$(ZB!(+AB&$glv*H}iPje-Rr{3l`RVA-@=pOl1g45@qhmHHa z2?m)!mH5a&3UB4*WjyT6*mnG(?A<*^#(ocXyGFi0xWffbASM_})n=+^kox`$tw3rd z&8^S!Hx6FooL9V@syEE(Nv9s}bLlbHB zO26U0<&Hkjcmh6mxCq%Od)szi*QJ?l-xX=`@W_5C_t6imuBcIWOp<+j5!2)H`?8bf z+HBg!ePM#%L*nOuT z=V%b!rTe#T=VBV&e8VozMF=)74~MAemKCL6EkKlZd5&YIxeZSWcuE0*akKl?y8Ikh z;CUny@0F7{D*A+>EhFKlPBNcKdk)Xvy0i4%Ga5_RCr1w4kV?M*m-nmIR;_cgyVyIKN-zdd}^KghDxmpbgjwLVM$VX8cQdM2z^{~fxctxP9LuLT;L^x-?{r-{G#vFC$@ZYTd!i90My31|-iPn|I;q4~IqG$WA`xdm{okbS0Ucvu!Z5 z#R>Z+max8<-`V?ryD7Q`*wM1Psm~YFAAA7ViK_X$22$sXI_LTyTLUL8n!XvN_HxPR!7g( z))dSZ!b~V=nEwJK+4-RGjL`w{>uv1DolGG_X>%Uc1qSEbs0$-7hxYw9H0#d zh#^8(6N3!4uAglz)BbVsdueyFNdITN&CKlQEf0D^=L9oGaE_0@-*6S)DD|*Ks}VSA zbtpv9X>~GRr2Plxyn;j=4UnpxC$eEOkRfoREzfV}xn43TmA|Q}!RVj34i+Jmy?mTa z-@n{CC~9dF9Vo+FtGt(WAtL8OSm@&fekOuwm#)~zrv?~&&O-umW#S`wz^o6gcJMT8 zf63TWra;wtt)nl?g3rT$bskZV$9$G=p2zXOrX|}B=Hywc%%fY&b2hYkY4BezznMKm zaQ#i{91GZy58`6DLhh?SON&Hyx5186CQW~kW%%FYrgOe{`R)#|4G}Uf{(#YTARJNu zl)>}*Spc_D5&xGNcFa?Fyumz~0OR71yG#lJJm%jB`dx4%(xsOXeLweD^zEAGLeoTN z{ih&ZXjQNL)d7oDH>{de z+j0G;6d95+T2r(Wf_4bC zE~FFsQAJfw>V?`t9;R&LI;2V2s!Iif>*dJ(?~GEfBDy+#-y>h*FMN@Stm*}iUDbf~ z-gTj+Th>`AXS3(;?=i8LIQOd7M1wZp=cgVmc|;sai{G?-Se=ZKvr1H-$k_`+Zl@lX zlo`sre4{@mLRm^~+)HI{j?nab$L?X3=U7G_=aC=Es2&HDPeeI#5_y3*v z422o2++dg_D)yfM4jaFA&kzPh>i@8yq9J7b+6nn&x=zX6-?a!qBMcMIm|^uw)D)b{ zTy+<*1{>f_iLV%NW$U3iG)%-Hm7xXBx@H!=%&;ke%VM7LP%Rd7Q^|MR!Bc2jgJfKxt z0xX|2BqE2A5E*D43lm!m+z?y&XpXd%@OKeAR=|1~EYW%{8+zT6Yu*W;dqa@yb~&EH z!UcNG>c{cr?HlMKQ5$<&f+*ySOJxpx3ji;(-`&QIv@ct^0Ru;l>RUIE7gOLQD`4Va zt=qjh!~r)_0*uwoO%|@F4P`>hFfAj>oq~s@7=dOG%RK#?LCN~o&_HjGE{@4>d0wtL zKy)kU5F}T*J(GldGoovE6dY9#I6eR0F;#EV5+X)nSyK+uvNga`cH$V<#{GD?!YQ0D z*W0yMav@z4dgUD(l`sk4#VUatV5-p15(1>!;1%fpRQVu%BLlM9*m!zR1l{e=mNWJh z;WavD%n`lqmGbT%$1Qfy@Pie)oHtt*wH&6CWbqG7;E- z>Q9PRwNa#S-Wi$t_i+%EVBTfZ{;%;gWkIWQUtt&@XYuKzY8eMqv#NMhFGXkvf zXTkc=Ct~8NtK5XMC}+`^^NpAv`ZX?RB!m_u1p#}YZ`pXp*SLmupZHdPb|6I=9;XKG zSkiC7l#jtI!0p2aAK%ouhDTZtG;ta5L_f|sxt#gye4|CZm(D5N94;vt&P~e2GKmJP zFj!zZCtie|G2-Uh*L=W9y(4+V<>ZgXgTk7Lg~JaXDj1tpAg2rI9}Gt5nZJe`@g%&x z=i56mag0-s&ot&uQP0C}VL;5T>Esz)++up^fzDW#3ZTSWtCR`WhL(_7RThlA^7Ws` zdA~~qDB-W_L9iyo?WF`>R3e;UrZ%VDoxAH@p}3A;o)RQbbv^ohYTsQt{s2b7%x7@m&t9-g``u$33)|>!wrn*UuIM(&x>VON8 zv2^%__ibY)N)z0#0OC~ZjE>t|rgp;yjD*b+dB4MMcTuTAf4g}sMGR1e&M%D(O0N9e zQ!^YMnWRGO-Ce$^%oCv0ZLSju4y>eDU9Z;NZ!Foz$#urfZH+@;Cu`{P&aWH>Lg3GW zTKl$xu3G)NNIlwBO^l|%!~TPhC%s4CWD;IB)-GN$n2yNGbEC8<1t=-Qj9C>WZSQ25 zZJa7-97W{)iW2Y)6Q%7-mmXadQS}woq5ZKpGr6V9Ls|(M zc=QxE_sSmQc60&#;QRUuJ_4c)Js=$;klqYJ8g?@|#>3I88Nl|?i;~o!k_SB$_L{?? z=W4aBSje4Lg~X{f+YgcUQ(FtO>)z!am~$JyB^}H+ppFV=AS~iVUAzeJ;cAimTy2NZ zBN(+!R(?to;sB9WQJs-+xxpg6+#`;r1g|-~4Y_AW`U%?55z>(leH5-q=u&?B1rYZX z9V*cUu(AvN=mJ_?u98w7ZqzE4W{ZEH6iy~bn;po6zVaeS_fdQlyo1X8G{8mjFI=zw zdbs0TTQ!36eo&$L!M826^L8pu(`=Me8wCKY#jt6)>m7g&^mr~XJz{Ug71tl7?E23Y>3OOEf%LgJHwm$3L*tf;@1HUwia<0=Ub$4{F zmJcYEgm+lu-bN=|KX0@rzRd9ei~)gsVS$1yWN)`HVeq!Q%!A}$XFHSydlNKkLbkmi zm+=+sNsrbdFW>!{s>#7xdFOt`ekI=TwEIK|a{SB!*U{>zH6ArK!|Da`mVyBdjBji! zp|DHrbqW5QG*;-?tG1r9LWYbG+P5Gl-+uqwu4_gx6i7wyeWGUiqlQ^!2x~hGi@x?j z=E=`@1Bcyi?NnU!AG~Hk>-=Y+j39{})3g7Cf<(v8y^({4rW_Bx z-}&){m@uR5dQfXI&WbBdG>6Xgh&Q)`0Pky{yp+O_lp~+3AHl2xb^u80+srpTDA}%P z-)dMiyo(QO4a#&k=-IKUNm#KGP)H2X#aa8QlY5caa*(NCbav~J+8)>E>e>+`1uk)g zDbZ&Uni9FX>~^e18y0_go=EfR+qg)KnU0yauTFMKiY}|UTN~N9G~D04!3tVj?*KHk z9i=Q^du~1;OPwoVdi;Ej=c(1>$7n3Wg*cWprrG15om$Z|)w@)~=71(MeyjI8M= zL)w>dr>`30I3Rgsf3B^`72SfR^&!-zj@%GzQnq0sokTmu`hFgF$NNpfR>l?$o)lp# z=I6d+>(?pyd6a3l0j1Yn?$wFHMG&@MnQFBwV8vKkZVR=9Ep~|m7xIx-XAoQKW zi0Kd?yZqe^55sFu^nJ(7ghKZQ-sTcI^d8+hfAK5fc>|`bFgjfPCAICf>=!Ne+8@6~ zn#ryyvj^G8jsF5zf`3NTs$ZG~e;%0_a4}VV7HiCW+)k!~kHgO=e_T?XB%U1N5XLZV zTU(T{nzmlCHM95zCkOtWFX|12 z`ys{EYelb;uPk5_rrtXeYBjn{t8mx4E`_yei_f2if~ehjeaNoiQ7`&Tzj_~oaBqeP z&oSUl*tAE!H}Zf z%s8z*%i4@IYD$9hGoZt9Tu>5Q#THM=v)^= zT}JAG2Yhndn3f!d?C?<}l(nf)MV%IDDy zJM=1EkL!@-i=nx~w)c5<(%G|l3~32dcU1;E7D1~9$vHi(sQySBxodKr@>yPCA23r) z3*u`RdoPP;0Q&)#lwMD+KYNcxOq=I>?TmY-tw&Ai8J=iQ^n9&W24Ove(css?d=0k7QVu=qtJgkXb)yfqFZDo5ZnDZ;Bb!LtEQ-P zaC}s-P~n!77am>U;AI|CM2D~}kaLHtt7o<+Y7!Rf@g*$8ckZ94y@%bOmqjwH zNSCub;8DyLto_vVtvrayHbv{>^(69_vqO57vfM5sidgOnlL0Fu++M9OiNV+7edsUY zklt@}zAb6lRpfBL!b0Yy_)irBmOHI@QDZY9M<4EwThB34i1@y3aFL;Sd?NC+Qh0;H6OWpY# zIS_2fwl%Cyo-6qkfgTpBX;$-o4_v&c>Qw<|Zq=RBGnqrnr`Rgza6ZWHLv_aCkj(8% z&9?Lr6L?!7bOm#8G1&Eif5GQ3w0WzRf9l()ShpA=7wQJ}B~>H1hX2XM{|W3*0C_Gd ztNMTHoR|j*h(IvsQ$GJ2yt#hHb?f?SY-AG z;Ku3TL)ufmR<84PwfbV=9mt;b{p@b5z*fwkKl3qW+V`i08{R~P`0Qel7UcRhjQORo zvWK&h(=(A~s7-oaMobQ;fm$xqcmm-NKo3m09&xD1FAKXg%+g)Qw3$@Bn48zVts<}!4X{noGNGS`c)!n$JH1?F`N?NfVS7y0IJSeHsi<+uxqDJ z+gnq(BHxv~9>_7@bSF5kr(oXfE;n|HY*fBl?s$x_sksZULnrvVj7C*;mTu%B z5?wR9a-!HQm#KDEiFd%tq-RY-r?=^RH(9Y_)#?*4(LXQ2yRc~9mR_Y!5Lb5gdj_EvkUyX;LVk~GsN#Ro;Aw9D29A6o0_{OsMd0#SW#8K%;nr2bwUt}X?7tF zt}u)O$(X0%(>?KK+gZEdy170S`LL(E(wM6D#{@|T-F>=OyhZ@o6?jSO&N1{OQ!RYZ z4fw+cW>&^0V?He%J@CS+@a?*C#8Y(mN9yF6;^lYtTdVGVGu}1BQ+vg?3Lt~UzJ$^$baJ=l<< z=C_f}IEdYRA5Cm&9_pZ=paj75I#ZzoC!F;VuEHPxfY-Wc}Dm|yw*W<;M+L2Q2eSU;^9QZ(A~jI5#eCOd1Uf4~%J%s`I*ifu+07j^0a4i*D53-A zurcC+Zf)cefHngrc|-M6;PI+JAl!4ON zi4L*j4UC?$zhWheDCK#|gi@8)I&fNwndmA0f~Jl*=imG|ZW{nzosZ0Cu^#sSrD-12 z{Q!55^9~>Wbv1jYQE#>guMxEZ`2(>=-t8ekfZhW%vV!XqV|F4mvk%k+7MO z*rw(oS>0J*B>Sr7?S~Z!M!l?Y168&@?|X|kgql)F8<^5r|_OT#|-5H*0H|nACK%;_A+q&*tyCfJEC2LjuOv87lo& zQ=eK?hBq@65KclpfkkAQ8E<_ZC^ApG?!*lqeYV>=g9P_28Zx4S=2~7ab8{sB_mF{1 z@7d|gw4}6&$%vcGmz!J?+m)r(Og_SmI9}?&pD>10Z-)!=gnhcr920*kv)}sXtyCI! zwza-vm=6{o2_*#b_P~mQAIdKPnFb5|`mxZX*${6roke?Qm=wIQ%nzx!Z7-(BioJ61 zz^uREHoJl~uf^D1$)!QI z%94UH+uL0qiu_1@_vXV_XwHiHWuYo3zk5+JEqVHUfbq*5VMO25U~z^c>&e`8_X@K< zuUz2tCL4BHLCsyy2rDEEyKl=Fv4|GT72a>?4BP>Gmd_pX8$YEQE2ElA7y9|gKXIIc zHWd&?rzpgzHrP4qo3|!o=P?U=hQ{i%@yS7+@UrB3NJZ zluk^h8#WzobR=K7IjvI+LD%q3l^JtWK@&Viq%=K&h(B{syLDp5=uZmuT~sm5_=v=e zh+>GMR0{v~#IxTnjj0`}f8wF412^XAR`u%7jK}0IsGq>lxZoP$i6@nM+qZ#1ofLC1 zPZWcO&#C)+V-ix*#ct3v#>O~(F4H++=)0pfcJ~ViwM?!W0YZ17+Hh=%x zpMTS9BVdT=#8qgn4H~g8rbmEi8^&YYAcbdCUWzvG$%9^Uppde%Ujv; zP95U_IMFq&DZrU7Q|pM(Gaj8v9Aeb4Q>j?}W3R>rd)rO>e%|1;$P5aV=Ph+*vRvCV#OwMXisXf^ug7Vo}M%)+gAFAd}n{lGAq^gu6B zD3Ot17@6u?8_dL7^T$*U!Sh08c69g#`KJ96pJ2*&j)!Q1y#a!|b4w?+>lEBblEv%S z%V5-$TQM%TK$Nw#XbIWrmCw}h8OAnMC5BkCpd}wJYTuZNbQFyh1f01Z6C~!YM?TFX z6)n~CuY&l6Z0H(uds7rIh$)2A6*Tw`00cVTxl$Gs{0xG}6RfAvt?ddY?^gFoACsXt z$eka1J!R1{Em(+y3@yNL1IgO&laI8y`oFz<#S|;R*}Z@41)cRc^F?~+@l{Y5tSp=9 zW{opd%vzOSztw=baK9eUaMRqw)*zP$Nf&CDI+=?Mh1WiX+xD~-A%#;l%k zcq^m(Xv(VvRN#{`|J=aL{vcjSN}Bqd0*a6ByI~`L&bL`_E%V98IIS;hSa_lk@A(Ua zuC21IO9WRojpJzBC8mOpV`H#aJPjCE)z0<9?M4qgMqz(+Y(3RKH5Y>xOMpU{18o1& z-;u=j^ourNKu-7r@H+v&G57hW3o1lRJ6b|T{(D;5E`cpCeE~w1Eq3v7@IU(hkwE+( zFpv6Up5LA;S;bIL^<8L0g!*p;x%zxc2>N$W3b1NgmilLfdjXQ>3-AR9HP7r~!7W!- zZ>$n*o|hV*>@VKOyfeo)n>OOr`=ybkppE_nF5iMs~su@LBH*$ZlT>lPXSX37+}HW zD2pT-QuE{Bg2dy{X5=0#y6P)(ArK|0W?3_jAVjRC`pSsXcZmW5cjtDZGAX6GitLzj2;cU1!|uL3#)-c zL7W5OtQu67B$%9)9k&7S^AFv8OYXRX!%n8jd50|n#a)@x4NjB3or1wb+>v>sew}L% za7c^A$gdw}t(GhUa#F$pvF$!;!zXL&LKCXm7Ujc{j(2?6YT0Rq8{%j*_aaGjdVRv= z$eC%qr6+7UgDpZy3%A~qnOtPY#^>b5nT5Q;?+sDV!4+900c@iq~`( z6jfSZPZVP5^YWhvc=5o1JcyhS+T5Z>l?$`5-b6Mn`e)l|2$v$89vMo+RJJ2Yc0;t|o-`4qVU9qt7bR({)y~-5MmvoqN9|3!aQA1kkH{Cn_e3$?;#P@T8`>wEvTtSW)m(Z` zxIZCa)^bLTPrTT=g)gcw1$IfGAZEXRtDi!H=yn~<Iy*D09k=k!u{AvFF;BKB?*Df#d5X znDtV(w-^<%JxZ}(wBoK@)x&Z-_iGi*Uy1dS8!vX1v+k7q?p=FNv)_vAOcF307_xU$ zFZhwnzG9I+Nk$m>Fiuamr{i@%1-V=|8iHpS1-52@LhCtrVwJzfwdX(6utNJ6(^rs$ zWyUAB;r?GY5+lozyIxW8a$PvrjT6A8##wIsl78KENh)_Hlhw!|o~viZt@fTyzt!XN zAI~+H!qeeY;+&S#jhNe66NA^t0f8DrD2)@C8A~`P_7s$p`rp9H!_ckDQ%)bztt73i zPhg(#JeHne7L@{Bq)yEc%0AV*LKd%}7DdDVUHc`_6p=b8g|Izdr_P4mPkI)*zIkmp zXkW&+G+&iQ*pAeYpVnxII@YYf;&#BZYD>!+9cdjDx6OVMdlMo9f9|9bMpouP48zyg zX>xVbm-|DQOw(W{HIu6Wl~@Nto5{&Iwl;V69M7kq@xH6}!s%OhcpzYu|zEsM^ zo%Ov^YN2G=r(oo*uK{=udg|7O%?{YP0ZoY!XdP7-uz{z_ls|!Ky=BWl$b1cErY&}d z1k+vb|9Vg`i-Ls1>Xuo!WtQq#y2al}!M`W3Uxw6`yEJFutOw;y{b!(AUK@(|z$X(- zN)W9HTArxl7%9^_SQdU;c7B%=gTMVH*9!gozqKg&hZTr&#J`C@?jHLu!T8xLud6Kp zfX?TP@6sA}P`~OPZ9CC~Z=)22hejSib&0P{b;N-aBIp9>V*?i1ZZGdo2M`{zd_~Dr zzkN-4|EIZ+n`klBoKoG{yjwh%Bb`?>P@7pZ=o5vRuFsXSj7idF;zkqJoz&!;znynhLFPsk@PUn&pEPwl9%#aHUqMFvX zxw1dI+!nvIukz3j*0M9D!xJ|>rVkBKnz);0yAbwb{y<(s`Q>Fh?|f0Z_Hie5?2UO*iPgy~_%CcvCj=XtIv*Zp&QOitEL z67IwGGX)XR6|UZ1puJTfn#F_kszK zQ|#`Y{+s(LK=Us{r^NJTxS<9xC-wr>8tsbv7bvU!-As1Q6;jNXVL`=MbkeKv(H}cc zq`U{QvgO`q2~GMVPb#jTl50O5($~$eJ)bY-t*>tu)$Wn%JJ74kws`ftbUC(|VFGF$ zR6JMyYySi7Utf}#!pGf>&zBeloPqT6ssX~IqavqDC+z5kd~QYex=c78La1*1xty!O zPuJ9J*TPJ<*^ReKU!|TGPQ-`#L_j03-U)0&H9{CeBHsg7%TDGzQZ>@*TFBY@PWbJU z=s0FY#+C%~D91-JM>Cc{<|xsD`~AFvERl7Xa)mfZHVGz_YJln(dg*k-r#TxOyKVre!%*yY}I$7q4g zeT%;@iu%)`-u%xX_=$ja2ahGBPJc{fUS9743w|+f+tGmU^_scNJ&EC2j+d*BtAYZN zk=8s39(A}h2QPJ_qyp^rxKP?O!kCR0%u`wPQT)B5hFA)oJ` zpl_d4$9+%oBL#VG6#Se^JTG&r(mUKjrQu3*L>9e6!`9gq6!RHd8D4`Z-oOhp&i04m zH(8geG{9Dn+j-MD(W_qT*2XAFpLU5o`%maM(QlK0=jP7yjOZeJeLVia0wEDOf$Aw^ zX~oTIOjIVJ3tZ)bZTs3IXYD=$nC1LYAa^hWnOJYSI2mO9oVs%r)L-*?PT4Nl3*$fa#{??0Cr zVXwqx!Et1A1KBzIT8#pHr%!M>qY#6AP{GV|6VB6bP{Z*2OeRcn=YLrd6nfz^A!(&x z%^gOuAWnts6ye;)P&aZ{~zA?=_-l8lPPXb+5tle<7xc(VsV4Bg9g$ zZyVgFZM(`~L3LNGdeQ#_WCOe_p18tI8mX^%c^)njXZado+(*6NvhJ@1 zE8kNyWTg1^`lh(Iq?RFVYtA0Z8MZe1bt+Ka+5c@%2=5rSVP&0T3i)k=CV=H_f!QO0 zUklCSE~jJus9GtL;hsp=oRJ1aOI=_4B$}*_`lvRpOfD|h4L)cg{rA|nQGf7o#vj;s zQp3b>+h4J70;f&Cm}|;Q?ouEQBiAG?4_hv9!fQ<2V-uI3Tm%EdKmJDlftGCTyk_m= z>R}0fLw9Ignx+J&7X^ z-jcuL=XK>g7s9+Jnfy+WpcDe7q7jwqQqcRZQ}v&N5*4ueua*uiY9>^{ZByx*ZO-e?Ay$&N-kD>F| zuR@zplOO*g_)UOzw(k6#Wmnf8bLUCV8EWYnd%db23#TJuJ}~7neXpObC#Zg<6gyvE zM-kt8pswu_0MKB|PG!y_wFWBt&ikxUyUp+O%POoTUMSz`RO!L02x;~iKTLw;oxDyD zNslUyHw$YYbo7lRljS&Bc1uhlgu%|Y-;#nTc8$Vvar9#^wl-S*%(jSnd2~XR@|4QJ zMx(fFSsy&U`=IP{h25mB*|ye&Yy*S zQ6W~Q@FO(ln!_Hz^-tcfT90}x$Q=lRO`qZAxJnB{X`I-!>R>y~|6IooS%1U7!Pqf0 z&1&CI2ig5i$u0v=q4o z25hq<$!$QCZNv&_2;4migxF+Htt78_X^8VdWnKZv3&38Wcj8jh3l4z{#I5xCkDU!$ zm9x!m&Qa<6ask@CT38Uk@S+pBIQv<;j$>5u*o||MUhWL|!E)-Cd$=FCdxw=-i}1=Q zO1bvCFRuG(dJLnAs;8<+-IlRtg_Nq4Vk~-*5q2zQwR&4UHH^7<+-f>`Mg^x^v^my( zk21*pfiKe^diBSh=6p10a-iy*=8ZgAk7^dx7DW@kK{*X}d%dpzjz$yW^Lm^@hH;drbJPyk%NV92kE z5~aWXCXw}^ltuZQOg%~O51G18MZAxA(%-Ap?E|ua6Sy&|F<;8TOp|-4#3@-bx#ova z{?UWp!TnZUlF?j`9k7evD!SDi{X(Rc#VZmVPolaKNACf5Se``2pByfs;G*VZ!}3FVRz-Vsvp-G#!gK1Uzm>V8*0>G| z97tnuLjAY*y#qB-^ilqL9&MFqbqPi!*XEDu9tKC_JKQ*H1EBfE=FU~k?{S+21pvVd zSRd%H?O3+({X8@Zet@*&AJTJU5uKx+VwWFj9cRoyJ>P)J+z}9WvUT7(e^j}9+8uOH zpQjxnk$e(~+3SrOKonD{!a=+d4v#y`Xr-Nmj55>(J z(b7b3tIQ@~UD@ZS2399zVVt4YOypxlkfQr72%MN1TudL$&VQ5KbrlNv*p*aqNz) zPs*tc>e4OT;AXRzNURuJ_gY&M@z}9Nb>Pt8!e74lU9R7Eya7?r{T3U~A_g^J2n zy`9a%dEOkNEF@@YX#UFmc}tgC2qu4oC^|HIqHp$utn$XEeXn!3E_}O-%pce2!zwQ-xneot2@L2wO_So^g|axA-vD!Y_Z1J+hj$58Dg|@%8I+%i zJOCL9Hp|HJ-Q|z!4zyTMK7X}G#SEN@l(}5`r*!$ghg085i5*JZ)Gbw~lFsZ_Er4zs z#Deu-|D*Z-kLLS7n(y6uZU51H|3~w^Ti*IVn(zN;z9|tVSR?=ctNEsE5GZOr5$Zot z;C9W>Rrhe-ZoRilU=y*2B1j-7R4ck7yrX#aZ@|gQa9QK}w+(R3h^T zjyPcINh&y8r&<(E5s*VQf_6`l{BXVoN8bmPu=_kow#FIv(NsX27!e9N)w-vnkT(P(&4goPw`4=!-FTi;s>O+6MK$fk{;Taw$7Sq zP7jf6FBoxSfW_W;U2U1EptKH0pX_|2vd@pUn$PkD5MDX#L71Pqz3<6cw6;mzdHm>YAOV)l)@Co-74BFM7UVUeEI*PR*! z0^I6t#~#IBp2l0-(b(7xBzv^qeqjgkMB$^V)X4NmEH8;LEj_DuX|79+G%<1NsxQG_ zD0-VbImLRQ*$DrWeuhx^VW-!@!wIC?PJTo zbdbxkg$!>{+aU%~fTN9$00aPU0vfW{^PhpL36`d+#fFk(>YWvos{~N97-yF4?Evgx zEL~U{%)(3lZh2Ph0Bk~DkV9o?8na^98rQ)5asMp^s`0%U7nVy2zqa0^TKcAkYP%y! zt=>FFeVZUk$5LM@4DQonK6a#+fcj-=5}^c&{t=>Sg7+C#*BJQrz|Izv})a7odyY}4?c z%DqO7YdN@Y_*%ZH#SubSV`!%T$N4M7(q2(LBs`=N`7oK>YRNMY>ALAn0#L7Wzq_$u zTWijTqpLmN8!_Lmo=s}p*3X)+Qi!mZ2QC4e=yfTuwFX%m1mt?=MwfDJb-cDRZf)o$ z%E*bgz>&bq^d#YNNTs8#{~fvB>Ps04H?jH1ZNzZ)td?4R$GNg&Q>yK+=9M@wcS##9 zmLrj#0>ka`u8h@OMf$9mb-WU4>{4g2MbhiW_4O+=&aZ$3&JM)8pspOa=q?rFf2s6P z&EV9HCA-P6JlMJa(}?yzs=v)PJ0~=wVtf?>GGl|>mSMIO<1osT>=Q=Nk>csd_YAx> zxyz65hjmi>5N@K5v6UUT(Yn!Bl);?V90#h6@ypY$Su~w5C;|qfzm5kw2a}MvJZGy7 zQ4~GKHmP~D9C?L)G@(5yA8V_zc7;o+Yk?L}OPeFgrNsj#jwE+=UY_w$YkxI|FVBk{ z_R<3`+I?z$lyoz&qWHh4{;r0IKk@|;g=dhE_lCJQz>MmT{jUAiR+r;S*k%w{6Q1bM zg#HYp{23}BM-F9rvpzycWYNs+RpAEAyHebG7c+e&h)6KJi^ppGpx3<8R5hWij-FGDoj} zc>@tKrGa64LEb^#H%#1%L`etGk4zPK!gNM_Av zn#Xo+{4Cnq4ucA96h?poW0bRqfP(d5#@hW;eaF+7FR}#LU;7mkf>agwDy>gDRR{o+pNB`pB%d$0OEfbJ2oxM(%ya ztoiGfvQc!adc*t`Pw{ZdLzD+TrD4WhNyrFh3T)2iorMg;Hj6G&`GE z6O)<3O0DkQPGN;EB*Z}2B#E>TnLi#+^S{W@W5<1F1=%|~aN?lHZ+!vPPh_nkf4R7E zIUrM-kfx*_J=?fa7`|C3HX4aRBNsv8H0XBgd!gBq#-o5b@^SEI+&kR^_F{WXs$?rS z1nNo)thf!0`gxIrExIEC&XnHqe+Q*k?6?lA?JYQ;a=QVuC{+GasARdVL=;DSKkYg>6xl0JC)y=9NQhh{<8qIR!#g?Si|g5NNEka z6hI{WcV(+@|8`K}|67{ie*%D-Di${Nd1HSq z`Mpcc7ZcdVwe z&gqK)`V+U0O?2=Bhc-JxDZ%p5Yr30>l_{t<$W0L#)vwpCaiCX4693*BzyyF{2Dh`yXaz0 zKH?DD?%^s#hP{wP3_&JC{)*sH9cw8&cw+UM7#0m<$(%baxmlG|^QBD){k!S>7qRy? z=0cs10m6HzbwXtywV>mfPVt{lh<$-})$?w9!ltj6mb{_!d}axm&W}@F_+igRyS|Wx z7ye=4oo~#p;5WNausYHBoLH;6W|S>ydb-$Qs7NA88UMA{+QryKX0LM%!X7^6r zn@*uM3P35Yn21F6iWyEdtcbnF%lOPVA>MBHlQokNT+v@|a`+i5B{aGkOgMk4TxTE> zR&!B6PL67_Xr8{U(xj^*fi*BnaP=mX2UI zHL(04+(1S{b!Q73xP^J(@{PlAI6J=i*-Qv2&u6ZeGtvq(oW_lue#KN1u=RP%$*`@U z2^oIhG%-R)U*k?h$1TG5V(tZ0)U!zB#Ja(gsdw$g>3ti}Jj@DUwlG7VJ+E!5(ey&W zjpnMMLdMi^RFwmV{{1oCV+c5nha-ejSSSf}bbsheHnmf<1 zCbqWUW2GqI78I1Wbqgwp0i}r)LEWHKMIcBMBT7jCg@BXM(BIoAVQxtzped54I&x!7vp2QRz_P_JJ^)xlnEMu%*02zE! z(xATcVFK%Sl-~oHvUz_OXNWhhC&*<~MWej!{t<-457wbJz!lBkc_`yzMAdEJ$xbZ) zQEnx5>^nAQ(kLG`bYb|;fJLBFF4`FJ2v)2nOZVBFqNn*kLk)D9^*3DcI9u>>@pF7W z$+X8tv^1n*Huceh1QMct(YB%0a>^QJ3}T7r*5if=B zQ_7-a(bjJ*k5dYd=$ZG09+}<4w`GSIesL452EU;K)Si`Hcb@Wko?1A6eec}szK-D0p-5GqfU?LpX-yLq z{UacdmQNelZUCk2u7@Jtlr1D|ef)s&UA^u?kIyJ0Pq0Hn&YR|$_u1KlM*16qytFx(+z+H@7CNmBlSoCBZF6T5Ab|} zGRrW;l3eS$MIDgEEDn%uwrd$@cn-Wt_r4;>`Cv5IG>WR>bgTx7)wfK2i5WDS=S0-& zf_MA%*@b!i(2J8r$7Hb1Sk>z@M*rxi<&LY?DrN+-(c zEc0@2KTpxK42jKEet_F|Ml8>9qfzPe2z(9FV%oy6#+*us1KtW`v&TrUAdAx#r47iN z-}-}h;(@DxvimB~cL%8|hIM@aAQJi>K*j5afVD|`64J*eGGMRNH+dvjHgE~htXERj zISHG&c;uJ7pl)Y5o3$0^l8PYibZ71>?ORMByWLqwDt=*vYpG?hKssI|}^ zK$N-eSQGGNAN*S~k0{?l2Z-`LJtHVv?9vK&G1-}(DRw;dNa9OtE9lQUMId!zcN37bsfLi z*g5rsuS{qKE>{X8>iNrA+E4~!m1dZ4bBa%e2F!Zv`zJS~n}u$9i1u546?0NCTqD$o zvJ{?e)1A%vD`;|MQyI(qiKOU(frO&Yvxq|uAx3tlA!=$uU1cY8vZzN_&C7pyH7=Vg zOBt@>Q~4Bk0i*Um6nR&M>Hcf^d1xG4^+1!Zzp$?xCms8ArZ5_#v#QK0z)_1vaX+@O zH0PT-w|in5d-U{zHdg8Bq!sGsOU|6=rgh7D=%(|pzRxSGAG%R$B9OGhAztv-_3(Vf zb7J8Q3VOA>*qQA7VsA9u?9Ef*T=?d0TMIAE-PLJ?Pg~lsd9?BBldxpQg^q|DwQ2@r z_ShYL%%~3lU3deyRFvy3B3TbaQPA1e<*Q<~j)8;IgYEMz7P@PMgaJtCit`sX_zJ?l zgRFW=2svr=qILHUE3pVojol@L(ptzp+Lx-qC>w=qC36f9(Q3ltaCXyYTAs(+2)X_K z%+*Yv)K$Cs8pKu65JnASiPT~)qrIfZ%CEx|@#}N#kh4fqdi&-1#$Y=hTLY{!qJkmA zFQ0Y*U(op=O>p4B~H3QzR`bR_wJ#7oEs zzKY#phkruf$3W=20|0`q>^0fr?%lPN23A;WU=m@FmF*x z?YcwC5x0JH$P^tg#?UYI&R(y`nZMmnv*ds-3`8CU1k^S0>3&~8)Ek}Tg_bGuKxE7i z4DI+#wFV&Wyno^YcDmI*$$I{DU*UTRvP3cwbk%ze*<0v+LPYCGy#S???J)Z zPWI=IcLsA4^n_A;4YZB^5uSkrzc=5XL>l&9{5K4&Tt+#}a-rO<)6y)Qs^uy%GVDD%&eT2z#1&*1w4X zRo~igns(Ho2uv$OF%*9ZTLd#+`R{}d)_W9sgFP3wGt(`jIA-XAWv

iHkj_wIhZJqa)^D)?B}&Yt6R?Bob$<8FGB1-b5DF5Meo?>XqzR7atq#Yg z0=iG*FX!dNo;-i>NF+jgH13umrt$39((eVK?gE8=zeAhmN3Itfg4du1e9t;dw;qEh z#s}1z9;9`}rShsRK&htngTAUsHOv0#<%wtlESr$~!upu&w2{j-KTRk#OSB?>{~CmH zI-#!7Kf`CU5%WD4vv>-ZVBO?Qi9$W^8!ewoQr~&o-|L`U6>{lYyMCXMsT8LeDX;@*cMiQGAl5bDZT`FMH&50xb3O|O z+opwnt~?XrF61#I`3j%LwMIlwuS?oeW$dHAH;E>fxjA17zCLujaPcjoTfg=cbiof8 zE0bq~g6yom17^^HraAksV4YT8lKvLAz7i_F!=tDgU z;$csT`r9QyMeBksc3NwQB{Y7!-vIk9B6g8F<@>A)C2^0upWoG5@>N?-MJUWf09L&> z!d2nXarOb7t8Nc7-*W-p@1o=qb_R@K|z>e#Kiva zss*w@`|($6Pex`*$Sg%bA}*S1YMD4Pn1tzw5w=SU&((p)Fm^q#aR7wZ>L@`%ww6G@ zyl5hW@Txk3i;zkfNUH5rxcr}h^O$gP87g_f2(pr63Q!r6ZMR%I_~qIm>t+NsOYqUF zFe8D0XTO&K#-FpAyJ|Sm#!>EtWZRxQ#riIlFA{qDC@V`z|1OaS?Bj?VzsPsbgt(KFwtMzf7x@WO9{Q*Nc*!f41L|-Sibxxlv-si7 zHRgTJe{m@X$1f0ak-XhpJGM9HsfgRB9tFWb)BB0rRldkrSxZXrt6KphZ9%z5b7CZd zK%Ms?-+up;RnrP!W<@T*1#Y#8$qA%}7rsJR$d}JJg?RBGi56<5wQsq<*2=a! zY1Ad%(`qHZOe{XfRfqB$=m7ak+bvA_s-mnb@-ajFq8z0t69GEJ0|zh}L4-5>UIO&! z3XYL6x4OWm>`8=0nxL0oWnwwyab>61X*aS$Z1MAmn48A?<{%VY((;uuv1(TTMwD;w zLF`{SZtvhUa_EpEjE9o&`B5*(itmv*Z#J2g(#qT zbb(H>ht2@Z4a=P`=2#*-EE?h3aHY#yg{zXxz?f-o@1HqeAYmC4IQE=hBhY8EP2PlG z-Jt#}Aywq96J#Gl@Eai~GA_ zx2c3_eWNHEhSi%O8*1WC#rHwr=^KB~JzFICT6 zC-;zj$^^^SO6#s|LlrFMBCu>!qSewrHJe}q(T`)^dQ#D2JuVB{u>rqD>Hwdd*4m6 zz+L;suKel1ynoTE&k!^QQXW^%f*vu2Q!BoA>}~D2dIQcOPd4JP7`@fbM}Vdj@uQEY z^3(w9usPc;v0hAA0sE=CdrRvF{SrB0)-(W>&uusUo6Cyi(n{hM7VA0OqbSG81@| z+eiq#N-W+*0krVEGKvDgrs&VE9cQ)g|2DvHXd@nta{WU5gY#r;6@u zjD)tlKY~3FEn&9hDeca{opSezVS>nA3R{nr-eFiR_x%TpIDP-!mtT%QF-K=gd!rE_ zgN?PLxmD&g-*9PC7*L>~eS-;TU4Z@)HpIir?mrhf^DgkN>U={C2Rmj8<`iW2sPl{7 zrS?rF)jaMl7X<@l0ZpS~TW+~AUwBW_WmDG-AxkYGW_UZ4^e!5Oo zDsW~(0tes`3%6^ZiDokI^i3!x%KkA3Ks|K+$dO)?TI+KpX{6AWvIf`mz z;YZkZi|pV-ZSx=)2r z@BwM;hSqC$bQ8+Cin_PR#2J>KxA@vsT7L_yWleol2{%7ozOh7V9g)W7V;6P!_tG~wGCGOv`|?T# zEroe(&LqXi7f%iEm3(a$6*TA6aWnSJ{bEiAr^ShO1!sXu@C*>X$7Yu*vPALF)PzcGMNb@O&AjaLSHq0x|R|C*nxh^hI^YT?(ubzt^t!-6~?bZ#T4VmKd?gVY&8cS zxmpDkg%ekY{LY|(1-8zNAm0lD<3L&6C!U@ymS!WKs*5p6n;RfiG)+h8t{mNmxOj}Z)t?n+7?$ryGxx)Q}mOZE`M01@La&aShKc>E<=Z+y587G z1-*Sz*axsL4knRsYkR4Cnqncx5v@iQg&$g|nMLylpd}8D5qp!hT%Y$#=17DW_G4@3s<2`Kn|U z5>J|!gnre`Jjm41K4g5lTU(%bYk}duN7tBw2AAhrZToL1e-)hWw{PP>bypOM?1Y`Z zq$$JLQ;sqeLErbkm(2wC4rtmB#RO*EmEiD87KW-CT8CBP#+q`X*d zFo4-iTmd$V#yrKai(^>XANT+HQmE6GfHEQS$HT4n_j7IbmG_ zXPH>pR1tpaYa>yDu2pV5s1Q4p9)N@&{rr(NJL8f|qO+h)0o)e5;Sn5rQn&T4chtc( z^|xz(Jf?Rv#OY(4gD0B(*?-@ZtnsjP(I8ah5AE?m>FcHjmYT623|yl)0%cfV#xK5V)Jsrv$Hn|57kqFjG;dD|U5)@b72i9q_*EM5P%?zm0;%{HW)*YgD zz@F?0k1Veon`q*OJTm-8rtaFDFS4Y=9p-yTH!_R@b%?b%cCha^w=P;x=ySq` zDawH)PfKKJ4+dj3G2_fpV4~o|pK_5BJ)u(@@Kq(}eot$Z>_?yGz$QQ z{z^rXtgS_PCGy|5?_G&RbX4Wj(5UG{Ce7$z6wIPr2Gp|uyVH+4CN~wFvNBFxC8&VsFI_ahP+@rE;r{|#hEBf# literal 0 HcmV?d00001 diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep.png deleted file mode 100644 index d5ebf3e1e7c7ba540bbd3e3368d6e9119baaf6af..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 15091 zcmeIZcT|&I*EbkN1VNhgX2b$0(nJ&tMM0!@0YxB2P(-9ds7X|s6e(gs5F%X^NdW03 zbi@KuB%y>Zp@f7UNJuil`+n}{d1uy|HS7DnH8X4G{UaymI=Rj{*R{{yzq9u~zr08p09 zy$fMyT_5r_x)}ff@V4!J_I3HZcLM+}JvF|1<$8$AGX7iAEDlCk4S62QhT7flGJBt3 zzQ6J(_j#honTz9xRU~s#63R>(-;aqMyyt#*>#nU)i)4$McXeG>W%0FEwMT=ZJjaU1 zddh=dleeC<1u&74-IIsPkH4AV^KEuZ(b96en$1Ls+2QM%yS+@}{qqvfuI2SrjX;^J zKxPi&cK7pBrmxQ4EFYWpr;wQpGL>WR%8k|hO;ITbm(%yB$LNDm`Y48e%P{ZHb=9vb z8Wur!ptJqy%ewow$m@E<&TB-UE_$!;UbDAfEscO+xYHv&ODcLUE&VBOIZn_Abm?jE z$A3-h*6p8QDh>UQ$J$P2;>^8t!^o2O?(R5ma)iye+yi7o9);c+u6xFZoVgTWgXq}G zBSltY`d}+elhMCAYj0H1mVFQ(5n^EYa4eT#U?6P^oD7s}weZ3yJL7h08ekzic_f&2 zp-&*0dH*ZpU#}TkHAbJii0RW!RKoZ?`PMxUTo)WT8CpbJdWgYPjS{z#S-sB@DZj(< z7sqsbFcYw~QXbHRWzXwuEO6b0QZUXa;JK7jyIuphn{i056l0m4ft?6-NX!E+$t{Hb zB}~&-nKu^EY6UwfKVy&V1mFnYNR1~WYPOyQn{)2`@X--~$b0*NtGaKW938NvF7NCN zhBkekI_)j{{k;wzS5c_zDFtTIR0X)B{TsnS%w$U=p-PkLNufKqfpq&WSSzkwCLQvtO*M}xws&*qBe>R64fXv!Wp2aFKsHIE9#0#QsO?O+( zIXPu_-(z*Eugoof?5&s?^HjRe8oAzCN)F7u8~}`QN}y6kCrIOTy-)Ri!Wq)_;*p~h zpQluR{#ma2UN6cwFr!@3b0XMMpKE#fTknPr(s=~!iVdQDEb8+tMHV51snD)&?AjKc zZv8>jOsHT}=7&j>-;0G*qWi1q@b@&nso90t6Ei=*_0}m!dY9@*IwD43 z*B1yUM8}#oQ(^QIpo4-`uyv5rzSF5{PVxK%agmCXnveWM185Dcqd9Uh%h~ zkm)wDSz+A!u0=-&)^qdvAP5{=>f-mP{2S6V^;Z+))xfWE29`7!(ZML15%g_;?*J)p zWJ@Qd7~9``qLTq=!1I1b}Ztj zuGWj+vCOuZET*;>;%`&`O?M$N7-Xgk^WCdbuTZ+O7j9>UsqTgNd)?jXa1_}E1ON12 z;37CUxadC{Y+|MVn(FhXD*OZ;KN0j7*4)44RgY?-nlPLHh8nlpj^897=9bSU-`_T9 z`s$X`B3l37;Id$e7-Z=mfdsMN@YM}_~)^^=3 z|NKRlv#=y%7jUq-3;|4OPr%l${eiCh^!in* zZ1u+`r8>`GAk~wqYm8IHTJv$P25p%)0NY$ET4oBwkQzQx(Ef19kSU=A>=ThlQn(a* zgnVMcmVPsI2t!~vE+Q2G6d;6y;|P~&@%xZ_HCk4BHiuUxNE?(5gt&`vB>HZ6@y3skVU=Q$s+hZa&Z@*`}js2IB4Jo>U^`mo-&nl*{5qw9vKZ#tHjid5(xI9x{ zzQ`vIjVLwf&{e|qX(v(Nn@)&gCsuHSwH>fu?IBnBQ&CF+cH`e}&xZ|A`b!9ulA>xs z=%d)D=*Z}r6oGgPiL3p&XKu--(8F@%e{s2m@pd#qSId2gIL)ywn@q8zF&&oPoovyX z!GU~q-1-x*QfJ}qh+AhGfJzY^T_r&1cX%7t6t_<$2_4yn1{)i&%ZC0CwgszHpvsQu z+>d%P$L;U(FX)Hd$nVVoWyqBHIcanqXdP-^J7hLG<6EZQYIZZ$7R~iux*#%sPL`SlzcC@l{4Lj{q?HFn- zjkowM<`DHf<@l{n^1lf`Q87hk^Vh!1v$oZFwA-D zjVQ6^{iv!?WbR=li1Q|JBK9keSo?d^VCuaHw0u2&M%Ec6W;rF5GZB)7X45*$(utJQ zVlw>s9s1su@ormTz3iz?^5#25glh{@RW-v+xp{KgCwR0167Gyx1~zKj&MUX8KfN=a zkmP{PWxIU(s@P5IYV0j3OucB8b3GqX&*r^lD zK_3CA>E2DR5Imh(7?9J=vods7tRey)}G5iK6g1^`@m_CHsaOwuV6MJz}h(epYC z;keQA0(=e7%-{qyrye5$n$P`51(z~Pnx4%EEsS=U(w;-H&bs!jO&*I{HZ^?8dWdhw zh`06iaGIL6^VqjEF(|3-n_<0oHTUvvG(OOG&Z)>D7)s{}@f!!vga)TEB92|Oq=s+L zrfBc4vE&5d_{E{rt?wOKdbrR~fvo}as1w!8!%N0o z?=>a@2?^}kJTX}gukj0{&-D>^?GJ&P+pghSVopN3Vlz(?8=6Qxx%7ttQ}Wz;RoXhH zN!(MXc`sFIH<5a^oVV|u<;bl6I1tUXl^UI;Z^RO3>@##Ej~|4PTOah;fFmgtRpzwq zH`N|1O$)phbz(jEu@k}@pY0Hp$-?hDwDWD2&?BYk;Nq0qO>^;W@yl)A2~VQ9M3Q^z z0~(##pPVGlPg*15^O4WD;Rf%-q|!iL1-$-Yv^-M;<{*+d59cO)bjdrE;LqIXPU21S zMz(77bf`D=@pDcdV+kW=lpji6`@O$=czx6a=SLfT6(h}5PbXrV?jUF9Rcym;2CFwd zmdf-gyI!BKUy#?bA*7;`ji0nmray-XYbKr6jFq} zt-Kdb@|_|rog6U3`Jr}-l=>W8?!X@4{6;&lHv$sr=Q4h}a{J|Q8d_co>?0-Ac^AP_ z#Q8wO@KYCleu2|9IlA1~UJ`GS!JOjL?y!d43}cD9PPelj^YNY5n$`9-vH%JnghTRq z>=C=MG3-qbT=xhOEjTn|zXQ?}X1ul*9TZMZq(moIb>b{mB}Au{7kwy7`r#KZ8RCXh$m*2-!DE6q%x;v8x<^T$;?eRS z6=JC#gZC)dhRN|SqYr+0o5^1_V9*G6bsk>Ry_#!Ug z3G6&Y?W1*b92F=Q`+S3K5`LTrFg(t<8#s0B^NLWOd&0_&;N~xTNWCJXZZo>Ug(w_Q z>Km~}0iTq-9f*O=*QU1I_chy+vh2y5p^ML^$3Jd6I&cvf5mRbCrdx@ip6`k{bwByj zGkqB%AoDVAUs5~Ee#v49E0gK{m)QYJtY*U}e;Mb0RPLZyP3ehfHmO6cynwTjdzH;^ zMtgC>-4QWX699Mqf3!qDvDBr1rcFlj?Ro6b$-+6TYLxVC_C?Vn0N)VXIR}=Y_y05N zUzNyluV6p_{w$%94%7N?kH@ri2k6<_m1j$@8DX!L^F z0QaM$Hx?m=mHGz-TP>V7U$6eU8$81CcHpCLFR9dG#y8+8IhR~C>Gh=H2L!D8kl%$?~aF&6idSi9n|cm_3C;FnfAgu(Hd& zJ5=x8X&2iGy_8}w_A~9A`OfRU2v`uKwRGqyIxXe|U1+|q0GKdNlI3L9?Pi=kkbNI4 zUNg9JG%J_@n|w}VYnEuDG?JrhyDw}y@<2Em!*FqJF?dU%)$OIX{9F-MzEgQeT%3$n zAcYg(v{UgXQvOhzpgtQR_ zfj+I)zZK4QunRNoZoqj((bxG;#}@T#&P*POe$z)g_u~PiNW>;L;Ek`~vRveI4IrT5 z&_BbvXcMnS!{5En^tE1emnXK5HpmmBT$Nl#7O#o%u;-f{fluuxId5Ra*F5%UQLYV!6tJaREog4_RF1CGKxY^J&Bn= zaY)cl#h54wE%zkabxM_j8hgrEvR{6!%A;(tLDhj6{v36>FCJcgLI4Vo+P8VtomiZt zhr6OIM7BwrYGseEsYSh9q}S;5qitbljL+l6I*PQswRgUr#XH2xS6krr`6ly2e@ZIP**N0Xt$g1j z!j8^bI-I%3!l0%?J)^izu~O8y;m$aZIOFaZLHA7h?zayL0`~{)g&k+1qUluaK$S&s zuw~#htV8n<2tX2GxuNoWBC$4yjV0B7zZL+Xby+RIK!8 z+`|)D1weo*;xp@s6mFOAR}oz|1pSxhkZOEKw{h_Fr~&iqnm6*zx^u&4<+DA`1Q`@+ zkid$mwoB^aUJkF<)^b6iB0Y5s`rDXpu4bq2z-3NQ#+mPhMx9k^Z48PIiBhV!t)=%B(51s}yS(+;8aV`F3C-R`Xhye@hE`kIkUO3&Hcy9HKcfW^Hm|f& zz>tIp)yo@Ohd}T#B@J8vEgYinz!_G>hfwrDdgHSy7zf(H!51tpmah6yg9Lb;G1T;j zZyP@+6Jf?(spFVXPSATb$lPx9AuSJFZD?-Q@XPEvEF5Sz8C-<_tx!RICjeD>gg)^i zd%Q+(scNV7qfn%we1Bbmf%+k}n_r$=21UqR`0-Y50psIumeyu*o)=p*EXQ@&MC8mg zeSNbGoksF8pj4dinRUF@Y5`Y%k)Pa3d)|Z{CL9mtE7F*Ed%0^oE~@9e?FKn-a9zaG zPGiXu&^izuQom>@a=k2D#}2J{1_yO2?QZqrs*UhZ)vA=Yp&^4`pSiuj7+YPtF$0SWkI8|jyv$vUTNWQ=)Q$J9cl>zHRbUv^0*>Ch(q(}DJF#@y0ytrU#hu5BDyJgzkU(NWijA5z3p z^3y^;Hq@C#;b0;CV`p>>e7h_UgUU4wXT*eK_v>0J^nZ{ z;f3ETlEZl@kP}}M<^c?kl9mfc$$gPk0JYSKh-%#W?rfS6Jyw``_B?$%5HJr;ZjC(~ zlUobB+S1#d`yqd6>uul8;mxoJuX|FZXg78Rn|i*f-KmBFX33X)=0*Zr9(NFmH?0_g4E6%icj2Og6w{ zHfs85j?c$w#K!kgFUoo+eQU&P_!N#4Szva*x(QwQLs$!Vy!=&E;%6QqyHmU(Gcq}0 z*=NV{75P9O0Ub17NY@2{Yg)jo&*#X!+Zr>T>Xft6^|vfDF;#tNTgfiTu<|_FIlcEO z8?)oApgiOkceihr$PJ1CpZbhQc|%7I;g?7Z2F=8^IaVe! z9-bPZRK6o1QO>O?MxB@67mhkGtRG-=1-j;S85_-aA;?{{f0v*uv| zf|YRR7=GIqdaEkYwnrsnW$Q7XH&f4{kS_QQ`bmh~!YvV*(1#MwueIr0%NPDe4^-r+ zyv2%Cr=rvFdHI}4$rl?5wzK4f2UCs37-f?P#QJi=dpsCqADSF!Brw%l5-T#%yWtWw zlN+#w4H*CjQ(dmY5-7Ll)r@eVVw@mB0q79a6xX8d77yF(mQ-Qe+unMvMrBnBKz*6> za-4>Y7p!3DUz^^?XX{|ac$D)aHVEUsA#R_2vm5(urMYD2o*Qmv51702f7*F`u z8K~}<9-R^(3kF z-^+ge+`RlB@A;24n57<+j^6W8Id1G@`OK;wKc~nGuNP-0V^B{kR0KM0szg0tkCR#Y57RFfytn;5*SIXsdY=Gz*vfrb4A!hYY;GoL>Xo{82mjbweEwEzmu(H@OaqLr;=j+^MqnP zhQ}+Jxr6rOXzDvi7wHuRAK9~S=9#F7D5#GDH zV!5Hd422!j8V({T6HPFbTJ39%rBu#QsmkOL=w2R0*{&AGz1(*fferuT^DAw8hEA8} zjw6ADg!KmH&5SX_`Xwdca^1j3x-OPFzO8DW-=)HU6H)$_Thib-xKCk2??5}U(tdqH zcg&I&la*8WBqi+JXTH`R08l8~d0sDgZTr%jpZeCAD%zqfV;0yNb0!5EeqDaxwiOo_ zNqsa|j=S_{?A()QplYA6-2w(pZCmcfhmPooT?w0lfc%;7kycvc`OMd%cQUn_BIuCs z)6x}>J6N26E|s~FaPgj($*}MF=rp10&M&xSj$5(o3#r+S5gCiP0r)NZ^IpCTXlL#=WT=M+REyXOV5*miqCY)T3sHL6thY>eopD6W?Jfqd|yEADM8?v%fz7KFUw7i^hM9Dp?=XubMh5WsES1aQ0?u?Y2`!O~y0f);qZ^YZ)u8&}FZ2VJxy=+t}q>JPe-L1g!F5>YVI)vl`cIU+ZoS-nx|A>lo;C zY3nQOXV(`>OGLIgl0-0tIa4*5H_u@oP}3;Q{@4)V@#>Jb=iX(k=g)LpDvZg}rRM4O zGfg~8R@H|dyLDVpK!5#oI)%kAc$%R{rmjMa$~q=Cyl5*eGA@>FrqB5Js#u1!11*T) zfgnS|y}znI4R>g$K2-`Hm5t2`(Hftm{Yk{I6RE>%lpoIZ(zgOj(k3{R>kroY_&3HkOhi+D4vbFI)pLK zCaiFNOlRp$O3>n1rLWZN_0alH@JsYl@82dt-^s}TUt}; zJGtOM?No|jxtCu~+O_ytc@1m%`Qk&mLbi7DW?-t5$LdP7<+82v2ZK6g+ZPe40@3G{ zGs-b~sRLsgCNm4Yb~Rfy+wtr9*pUI(MX*8)AiM;D7=zx7c=>)}SI5BBBzd`mY&muG zn`^meN>_UL4SM2xPx%8NZJkO*k5c43%#>6OD$?!o`RJ7J+&NMzcuHb}QMn|_HM&c% z5}?2LJd?!pQH%?;NC+MemE%&e9c8hDHc8bO`lmmuog#Wof%{-}lTpzR>7|6Qp?;mt zp|n>~QJ)JSUFUzfk02%z8pl8XVjIYYbdix&T5n%eIglYTmly51j2Mlk8RQcf<%DSI z7NM*f{)3y-IZ50dGCb_}?!1mxpnm&$^RX3S0Xu*>pydl!Rxnuw+b6}}E7M(fygnaR zs8Z~;HjZ0rSVS3+kkN_hU(gmVwOk6l;KA@kt6^RR?=DvWFx?V zftXx?P<*Tszdz9HNV0IX&cBoH3Y=-hHQLd4>S_i5MwCtY-lX`(^i9eE0N#rdIP*SrU$dW^CdwbGw zh&jevyNNFsNgV&Wid!<*H6&3hT-iaS5U{2WNVd{)4fG`FWor_R@4%}k-(M6`)8o+t zvsGeJ`FHv{B2OUtVCbKU+3oc9w_a%m2EtO-A$Dc!(S23~F{Ovbt47@K?%vK1ii}O@)^3*EKM~~1Z9i>$ISB0rTU0*3}N0%6< zoQ;nsq`iRmCC4jqM)N<0dRXb{9Ykv1mHBX${c9Nv{kdOCgRt~*f=Qo1ZlF*$QCYH4 z(q;ifh@q~y5hE&zO@^3wLxcE5?n;R3_=c1b%a8MIfONqOE~~m6X8CUZkgkY=PTMS3 zmUnmbkt?E{4PeST+v5X3IGXiY)S;t*%>)aUhssym$cj$^0JO*dUFB_d1mmJ}@a7X1 z?{CZJ0ps#shFnG6n%;tp_(i8McMyAtb8_Jv-+s$KKW)mXsYCWPOFdv%t#i(?LT+x> z=l+#Hk+2BsRY*m6l7`92y#xtI9w{3cBqt|Vx0gWo9{!dvnyW8e(% z;Z@{n8A$`%Q@4Z-Ydh(4me^27v);&<5IUM81X)Z^3EoW-F7S!9d) zZw}DS`A-{1`QSbFxn1aiDL2Y||92qAW!x}tEAPSX%g)k94ssgxN`5wf?#9 zvdCFbjB)fiX_SU{4OsZ-*RBP6d2T^`a5zq*37s|(yGW~g-HEo1;Pa0}R#e(`X^|Uk z$sMYHvL3e|adUJJyqHxu3RG!dVvu))o^&!GUD;r*)91RPj@=nbn+abX$ft_24&{gv zc+JAt%xwwEsJKUP@-7HHVC3;!!7@0R1S@ zxLX3^UL1Z-5W*oGB|UgIdLlZyW`&SbusgDgDqAEZQ&Yg1r+!4eY29AdT!hFaj$RZf zER>It-oZInI<9^a(s4$|%KE4;IB?fxY|rS}ZEGU>Fr~?=rt2Bn4%aW632Z_` zhHvP|NB7GJAKXTC1xa{ylPWi+#vYn_|DEPP7re+6h z)%9n;uRyYKl5^b9rr1}cz(v{M-U0r1E@||_C4HHp_YplrLQAsoRvui0e5w9R0-u<0$_w>mic)@-|& zpSRH3;GIO$;7)wBvhsnxP*}P=@s^^BE4>hMMMP8bD0MxXLL1kzrfs4Y)`h5>GiveC zKmGSzQT|BLj(Y5?Jn-aA67w?4ghsQ1-+ytZUvtvECq9wfs%>UB;gCnwl+^HEsW@G% z^L!#9hc;=~n>9Xjq%t|Iw+$YZwS0{D?if7*Q??Ojrb*mrACRd;abJXBw!xQJFxZkS zLzMTJJT$sCQ-KzIcACdH=OxMX$8B66S9HIcps(Yxz*GhEaccqdUtCT*K|x8MYu;lX zz7@tkcr3njQR(tr#u!Yc>%;oOGF{VxhQwDEmwoYjF~nMe?nfy{U&oWLII8@vlR#;? zHz@TMx6hbTf3+x<7`lFzX$EEzCDmM2kXgF{`Ls`P`!M52iHBi5ACMg3zz@jH7ZL%u z;1wN>NMwh{71OM)Wy3!h_i756a+o76_81@N-t;eof2E>=9{g5q%A4F&AKR{b?9Jk5 z4}^j}9z)q>6~wlJhQI5uq&(F!CaMMxPu>kNFXNptCPDn2^(it4h_j zpAuX<8r8@Sp|_w;)TYIskm}8u+67vCsiJrQ?q2B+47e0IqD{O-9lwS&P@M}!s-0j& z5N2ME=B^&b$F{nwNa}yKv6f;)!2BK6 zP0QaY<&z;u9(YexG`d1MO8TrEm8Sf`oEffk>v^=a`eQhKY0kZJ%pG=GuybXGd50qfkFhi$=f5)B|GRDW=ac^)nB_<8c^v;OihrE1|6hvY`|8jw zpM_KZSXXnC(@z#HW$!F%&y==gB3T5SJ*RU|Syb#K0)_rZfxF!fVXe6=#tSSnLw8W?$gIv1e)_kRl>B%|sWEsM zU@_`w+1eL_m$%+%ZcP`s!1DeJr>;Ee54{7_n6X*B|54smTy1jhe#9S|oaOH%3WUHo z2f5p+jV2l3LW730w)PnwOkvSyBiWDQv3aJF87!(%RV+GyJQjpaHh+u>5z#Gp8ds!y z4tMp6vHIru{9U&$_I(4>mj$v@oD$X(W0Tr87f6x!wj z^A-N|)LBP9QRu8@8Z3;Ym4o!_0AG{Ti100r*SFqKf6aYGDIK?Yf1)3I<3txvY}Rb| z!0Zy>k;m*nR!$L1UkNVV3E_?nFB@*lu?Q)}y}qOooz<4I(zetE7P?DK-ypk!?1Z4vl=-|b}KCsM!l=e9H z4BHh?7AKca!tB-jfR89duCdAuMrP3sGqu|+YPx2=ViNtD-^Z~fti`k??S`3MDANUe zwy_NC%wCj_$_j56o^}ji9}(WLN=dU)5rIamipaD{yVAx$>1X@g!<-w>mu!cZxwB@d z2*1e%qR;oE%I4nF&lTph#<^fPETMqOpM!!~Wq;mt-1qTG=2uz8dB{Y5?Xr^Fd?D4e z)dD3_($BA|y*HjV(aq})Zk2M*D-u!!9p7z~u%X`_Y?0aFGK$Si1qjl!vf5&^=B2i3 z?&xM{UZ^_=xN!(VXfn!va@EAryuVpcnxOaO!Ebb4JG++BSU_u%KTc;IEeNg-(y)v(yq)F!sOE>h|j+0ywy zCg<)oweDYu;fFptq53a^-G5N${^8L5Q~7@jnNE@igjf|tTw$Tv9#PjvlC>89*Ur#l eg78d^MYgf1;rw(f)*q$-#s(HwOZ9F)`QHF}_P@gb diff --git a/static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png b/static/images/running-emr-spark-apps-on-spot/sparksubmitstep1.png new file mode 100644 index 0000000000000000000000000000000000000000..b0911374843bc9a0075f796fac85cc5f2e7270e3 GIT binary patch literal 14975 zcmeHuXH=8lw`M{w(xi$MQ7MWv1r03`L=jPnNG}0F5fKOmrG`WiQ4mlOL|O!-C`tgO zmqZXEDnUUEolrv$HGzbL3HtlDJL|4lGxy$^4|6|o$lK0)+J4S{p1t3PHdf|B0vZjDrEZ@lZM=;eR`^FnRdjv!lyQM(uz5bWzAvN1pb_9 ze>y3ZX1FdQn|LPw$F&a{l|x7t8EK{a=h*eS0ci`5+=}z?YX7T_!RDJqtYykJixLt0 zSFP5`ll|(=Jms@#%zf8A3t$&e*-DY0$ zhcG7UyMO$65~#?mX|Vnoz4c_6R*&9#+kglx?X+h~4s;e;k8Cfl&1|a&u$#HlIz}Tu z2+J6vtetDv>RTH_hzEu+mh7)U%C`o&vx43*+EI4>O$}RDg-={JQX##CB>mNb&2)Bm zlaZUxkw4Ubblp$Q7lCAeyOhZJGx)w}NY%hRRk07Xx_rV5i*8JBnxlqqr_2Po<2H@- zp%EU)ZwWJd8=09i;(KbQrU6lYQ>~GUaUJvSCt?>CXU?<)W zzD7RlPll9zB$+RxTfpHY*adEp_sP3X1}88o7Ba2P_{0`s6gvmL0#$`il)lg|q=$HS z_RCdWf_-T_`Ma|VAFGF|N1_dx)M{+@&#uZ57Ik&zlsp$W3+yUa`R1pcPp+ zrOjQ4}BSgQ0y(dClh| zvLY8&DVeb4QQzD3Agd;+e#dTnyIn>x63b-HjAMH0_B!VFm1pYIFfX_&aQ<0hal^N_ zw>r0t3Kq*bpn#lNOVO9lO%_h01a zixhUQEkm+3LpE%HWXqwE=f}^=G23H>_p%@G1wSr^iDg&UY#kNQU0)X|RJ5lrdo3^^ z6_|RPHCD)ScNFrs;jqo51-keX63v{0&Pi6TXLDG}6n*?496wMfHY7n))2TrGzLq#O%L=8cG#q;Q% zs!pJJBUOB$$bV%k@SEb+6{DnL>pymzdA%Mq3V?boB=oaeII#YrDG$^2d~&eVHuG9BOrO$gr{b z?`RUcgC=yk3#RO^)$SbVRcp8XXHw1#TpVDoJjZ7JHrMWxsFbMGmh#q zwsCB=K(=E34ZfKCG4iT{sz?TPVQmM)Qe5iV4CdaxC+rlg$HdUCMN+Z%_DtURiUeWM+Q)n!(D#jcDcD%oQJl@bCa) zTTLSTLtTU>rE@UH!|H(fo5j{?_}i7mn+Ey@FEmG9ORnncN9I=#qGs{y`UwOeK!-zE z*4_HKb;{T~S3do^UIHd@mtu8rl7qpZQSt#azfrj8{A4swmd8^RCqR%7bS+=7*X=Dr zN9c|@GqX-mp=p!(veh+qEb`MT^J6)oZKs$WgX+fWm{S`Wr`96F|D!}j;Y{3srjgCjXq&yrd zWq5#)Rqkj*tVnD5Chh1wY%x|XDjjhFG9}yWd*&^s&O_{)0GW^3Hn;fxVv`iW)eL8H z%n@)ZLGZr%;iltCd4;hB&WtI$-kjr9SC8Dd1C59_+K=TR^ z(~&anuCjp35l3u4CNn|?Fl*@|h!qK1wNcg9pe;%5o=78xwd<{-V|3%#F%^Os#rGk6 zazJpLx9kv?+~5OEIe4y`WN>wdu%rYflHW8u z|Jb!&NGWjcr1YH+QY8VBC4s!zHJ)Mu7i7U*riWeC;#VVbMSJ2eM+mnz-li79=9}O( z3k6k2wT~pAzt4+^O%47qUi0ou_)vS>{^x4I>!s)#oTMV00mRQ&M>TTE0p>(u-`k>x zBqUQO^c+tjo_uS{JGTK7i%-{T_FJx~fU8-YJrq=u-6GX@UChI8v_!`F(Q5M%j>oeH z_L#dBukP2V&YU}({4rZqb14{^;H{qJ%9AwRd2=L|QLz}&!cgh_*WuDV%H15x*XQ;_ zoBLN|slbIv8ON}vp8H+jiB6KgW9}DORO~yfPRJFEtTLYQ)iS*QVfsw$NVesg2Iomx zp0&F9V_IFoO4Z(K_fPF{Xs(-uG>N2^_rbr;C2-hkkbKCJ_;+<4Iyueyrr2czAa7U{ zAE*XschHpYtl<5@9ErC1^%sJQa3z;j;-|ze{Tn9l5vR>DM&`0NT?D=QtwS1**$sAn z(tE%GPDpYj<|RZQ0?9ZlRTEw1u=Uba*^Li5f2{P**OC{0Y|Q_q21Y97&|hzbm~HUI7AFdVoI$f;Nebs5JbprEIVtP~0&&bUxR1;q8c{VE*9q*9 zD_2$z8-o;%atJRbD~z@Jb3tSu zfTYbOM-mo;0X%nc2*cvo@6l#o6B+CYxgDY_-?1>Ch_I;7(L}!?S{Hr|r4~d!V8e~K zC~OHAi-L!Q9LMO3=#bp3$=0{K1Eu?C{OeyE!b`^!{OhB6JKUBK+*ivvz^tk&FASlt>Nwf{Ymzjwb`utu6)Cs)TVXkG4Cc+1dfec_eCy^IFS= zOOK%!idswwJe-7q)tRtRqin2R3A@R`2S}tlmK;eUx)!+bQ2lH(M&A^NtyJ`l$cU$M z(j0bjf$U%U=K$oL?T7dOrxc}_A1p)X|8G+ua5WK-VFJ9zHh zim-V*8L8If7Ujd=5eWmp_yCDmG(w?8Fx^Az2-WKD_Bb9+vq|dmWET3C+3Ln0fBpRKJg}}%)DBYOcm6vR0YX%SWSCl`$b+x z^^*h zw0G*9H2H+E_sQS~E$VHDDANrr-(rM7f%O~lZ(5raPl|`!Sk{mjo_=^=^Cw)1xDY0+ z#o)B6f<6?|semF0tvY?8%8XwO>ASHT@4+6?u?IEWy^p5pUOPQroaL(SCXR}WL&flF zitK;$vKI5E3zbqig%bZY?y#AS7TD%2j=%A``0JQ|Oj}p?Q=3mI>mMF`U5FHjpsyWp zf$FG)-_UdV$&%}XC-$IHpe^z^1(1P?3OK zWa$rzuS33#_&QcwSb2sQ*J6ePF4}ByPIH)C8})1bCNM}7qH^p!W13dNq<=QOo;~h82 zyI+=l7obf4|FrZ(X*F$JrmuBd`?1;AuiDp}w}d!^wNZ2>F2Gv#?Pz5ZfK&8ep@saT zE`yMiRG4pS80j54GeMtxxJa9rtVB$D-6Z~4WgXcDG-X(Y*87nMX13y&=(}A zgcR$_P&ToeM)~wY;Ke1(R}9Yd^FwN? zl4UlfeI~>p?no^JFnC{ai^S^{SaG4z;S?_X!z`OB%G`$7){JAWn~E-t;<%H}c?N&P z@ur7s(I!sQMmZ7@Z&J8(o{4~{E@r5s-fz~lFpWp2U_%y)k(93nt%|rxYx`e3a`NZL z%Er+g%H;SM+nI+y;z`_qn9oHZOY0yBb_Xkul78k?eS0T`y5yzng9epvdZ+bY5wsLFJ8E$y>8g#*Gwid9cAa)#Bovdl|%!zwUARPJ5GWIj$!Sg&`N zs*0x3J=AD(?DPSC(O?-1IayeRf0lilT&ZCr0YSTx-7n%P>y$OUeu^XMWp(Y#3JQM` z+kF4IQgw;>iz#=SV`FM+F^^`MHQ+9k9fB?v8!H+ih0VDrECjl2902yza3kJSjBLv& zW&vhNV%vJb;@Y1a9rwe`a$9*MrD5^b@WF_cTTXQtVXMGw6ekI&L^5(=2-ZB4S~cQZ zqx+AUup|od*f_MTCwbRGw+)6^2Y;ajwy%hgl(uuR6G3L~4@C2;-n`sjUH@2@lH2lq zt1j}UI7OXcfdej~L}9Q~;px)_GLMd5{oDRN_`Mi4-jruBua1tp|7ZJ#o#Nz3Vtnx% z-8+V1-~@?Qe`ZFhZDeY^#q5B!p7i znzXa(2{}Z^+0}@;sEDKPP47$c8}u`ia^l_(vzTsie|y*uXWQB`U8BRL!l5} zUOCCYpOp_#aH!6om*K~Efhd|q;`VB24?=-6Kihdoo_nqu!JL>Z28W2MZg zQ7Y8hE#dB$oM4B=TZgpo>y)H5_>MJVB^a&Vx4nZGmPl<}+%F|p%@vc^_^s4DD=%lo z3XI%z&4C$vX>?S|iO}=I1$dL#5|_ReaUM>CXTP(yVS7s5i+$?#_gmcmICa&TNm^a#=Z@VoTXp?c z`*%Q@8fB0+fX*5~=bBHg&s@c1qq_0=O!=$^(YEchMNUGqUCx8Bqc3T+SlvZpHZhKO z1R1L_-}+tK@E~a;xW3G@bAb{oT)&(kixCLxz;8P?fw@2ltos-e=s1Kjl&Qbh`h%y zeN6TbL!cwlurB7i>FBh!7j9p3P$?sV$}{;(O|K}osI_N@TlDIzY}&EqjIpAG>?8LJ zVPEdPj=#bjS!=Po7gj%OE@fMu12!(xINWvHt+f4eKzwzS=@X-DwTle275VU`8P?$P zic?>aOtLH`%AHIHe~U++X=R)s0L5)zzAsW(cJkL`FfyRpZ?q=QN=hq$w@p<6ht2GQ^^ zTyS$_feUlyk_DFk+=ZzMwS#jJ`%ldInLJ>1Af+$^FuN#$e1a`MC{5F)z`Hfs|x4taiBUp54JOoh#M%cg@(j1TE2R2aR z1hHoGCP@GT~e`yU&JwH{Qz_@p1fM;0MVdRHi^)DdcD;3{5c z&;!^ssafZnP$_%~^_KTDq({eK<_F|zbia;SD+sCBwX9h(1XX4>JOHojPaN-#I%%^7 zUUvG)v`kiS#3;`+b`B?;)9$0~aTC@Rq>W1LbaoBvlXcv2A`14|W%rS;Dx0&Zp!4%0 zJUy#EcT@txaTb~Cy-bdzfc>a2aMZz$5j1oB5vFJGeaL*EaR95I%9GU(TAb!j%6KGb zV%jB6XEXG>_F4*GV|6r)!j%L}4pd=kj_}-n17pEFbP5Y#N~PWGUO3d4wWJEStsyOCH+WOPCSu-ahGh_e-c&e<7QeNfw|r*N8&|8y2SON)VTwyDOs5h^N48ue z7cdX;r`_fK<4cBo?Qt(^ndfqGUrenEtb43HJor4eZE%)0URN5I`K?i;yhAGFt95~VUTue+MTTMtW^A2 z_8iwb*rmFuzo&R$0`iqM7(h6RUJSM?F2m0n$$1)*K1b5UuW zgl{|_VIHpEsa#p}3d*rT+hN_2O=nn5#ZtqWbVlAM+KCac*#fJ-{7dAkn1_54SB*bx zL>JQb%SmU$o7x;#ZxvBa|EvxZNdw1lW#MB`b|$#m07ou4qmzX=M|2rb==Z*NNfZ{i zm(8S;qc6fb(LAJ4FV-~M+24iZstjiNpIS~8_|DJXBL3HVC7THW&f5T7bdYOo7l5bt z3)dtoa{rdzH4;@6_Irc9arti7w#kR?nKd?buerdL1>Sk4fd4Q{tL_WdqhSr&Z9kAy zv`Sn?_UPS6-Y%&w*;X%)d>!3`1peBoG*ARsB*JC|Ck;$yV0K#~%`X5?m_>`-+e3FD z4m`r^zMJS=>t8+M=`k#6jytk3ds2)PQ3EG&Gdmp;wYFLGd*Gf}9MGPUI=!9xUR1kc z8}isM8j5A&wq@KU9hgs6CAyCpxH8z<=|;lc?>VR^ zhzpYRTgcY>L5d4od#GZY=)dE8{G0Lgyvy-Delvfk`v7qXl*2bggeVLJOn(XZaQSaD z_GdT$-=3=fuKK^n`rkOd|MdeRygnRBnNj`+w8X(i&-@*GxIH!Y1WK?y@V^@J>ugJ6 z^WS}Q9N4dBG!#bP|Ho^0x+r);;2#y~S^jfHFMXgjw*Rw-fAWIO4h!htVFCXU-QwSS z))OVI@kmd)5CbH27rWjoVB$fW0!a=vjq`Nau1mztU*kKxM-ln78ac3tg4rYE- z+%oyaS`yiY(Pv9A^B&9{6W`Ht--ifavJN~q+Ez(HP$F-4M)QFlmXOW6e>}Oa+C6E= zxa{cI{bF(Z2FuQmIaLbHIj^dPS#a5A!P#M+pX(u;x((*D89fZC(t430%4!QHe&@B8 zbPGm>bH2+=o#oSu$al%5T0`m3t<}Q{+>h<8| zVWiOOF8!|HsZmvhpswxAW91hVpiM08tIJ7!kH=_eZ|TlWWo zAC+uQn@VWgf1hW~_Kt_UI;aIWfew<6Na?wn_0*2B&I1z?Zq-7>@6F$mj<4v?asMh5 zPebeiEgS1#hH5rK8_nB&7utmRKqqR12NjJuldfxEADzc7PuKxBee0d!r*(pQ9*<_K zKlPI-ey7?YH4I9xx!P8Hob-$V${@^>iz0-aN$@WCgwm%EFvwSO*+94FSVj{sn~S{@ zK?? zLH3)~)2-*^{XN+HWsG=XRFKizOgCt9Km$^bgW;=+%*s38MU32%E zpr?a&rHzEU%#yY2({WQJA#P_Wf(vdL&rs!`Vd+RabisDFWy?_nvC}~#de4#`9weWo z+l|fLWRUNuHjOx}41bIC1?n0P4?X`5T|n^BG$DoATcf{Py*01bX-Fj2e0~*+j6TgJ z``%arcx!&SwIsp`#(h9g;0>uR`uZKSG9cN$KiLJ1SAe!;pvK!4BQ6eHDtmyjhOAo% zmem(k$n-DIyu8vWme{U9?ktKqpR2eRM(7D&pn|^lMGuwVyb|@e-WM&(oiGs&q+YDX3(=(_e#vNWRH*!_i^<%Us4an|L`w`WsEJ-mw{)? zxkuIl7)xGC9@dtwAaeDwJr2pg%zScG*HB*J*Ur1QmK zbr>{QG0(fBX=V5f**1P$f{0-xj4;;dV>A=zzBtLn(A~SY7(t(CCil|k5uZ7&YE*9z zWkc$tKT_v9{fNXzlGXlDoGsQd-^1>m#I>}35#+|Hfw`j`Nm8tZrk|#{O@}22h*8g9 zI;2+v-RlTok=rK*k6dUJ9A{CjxW!d;bhussR8e;1p@7#pNi`kk5=6?~3BiQyEkv({ z!2OXEoA34UIpy(rV(~8xN-sHGK>F!)mfEhpZkxxCW6EI*wxheptxD~Gct;$iWe&L- zf`b(s*W_%m6;69?b32F4Yq2sV)fgNe|B2>QhL``^B&VmMX!Z=WYl4zlf7`HX+DR2F z;&EH(=Ey{ZWH?rkl3P^WO0!NHLQbVX#;F?IaD<+o;{x%=pth11(7-$^cnNk)I9GpUx*k$XJw@=$HB}(n( z8&G=FT5Z?e(S+>&B9F5PA<#n6O^*HXz8-2AMSp@MYS7Q2avZ+oU-m@tBFHcrF)5cehP(8}n0Y>75}@QA4Eu3c`BAB?oY zpinDh)`&x`lyW_f{$jJ}{o8{Hjl&;$PIcV@ z;wk7VqsPyx94TH(y0Z=fr&*1b8I%K9trm_XPk{?=TX!Qq+TdgFKIQ=)d-3(MB(pi# zLE>ll)cTrqxHN%sC030UJ6>!f#W;pJ-5PK;{z<`)HU)Ubk>hh$*7@)8n?6wtqy*ry$zUmCyDh^m*_dwqmaU=qhtkj2dko)b8n+) z&*Ck|&rF$#@;99qw^u!EKR?7Ur-0@br>vrH$ogz`XH7V**X_Z`6g7wxpz>cSnKxLs z>WJ`x5@OF-%^qtE3cl!&QkA~ca}X_|uGh{m&1XoaMCW===k25YpmG6D(TmGu^tcYqjG4MzlUy@0-PBXsv#w4Fufog{OI0Zs>VmeKGL~P_AReLk zv_%V-rLW9Q@2BOPDtHeH$ffJR1!zUCwI<9h@IT6;>(z4pt;J(sF^p>b(c_U)_m z$kOjKDy8pW@0Qb_UcOF=@eeLOi+7wR%BzovM80M0HG2~qX;X{0&`APIcM@8CKR$q* zSZNgV{yBb@l4z(>OI!pSZ4R$tyBOJWj`aD>fJK7}#+AkTU7)E$-Z!(Rz>3t2p5CX? z*67%|PVlkY4SjPp_60?~vq7e>i?j=tDlo5$z;XLYaue~W82fXOL0Y^fM=S}IY1tyS z`{=x!nxK{PVY9^n-u%=lTpFg3HJ!}5wmy<^P7Ee&iu-6lGbT-|0@|?;^Tjy`3Nf5X zQDZTNaK)*UZ0d$%3V{}vA>3gy>`UVuNt<<#icZjgK;GVB28)XqaE9&D?mf*`-_yll zsKkqqLAH}hO&@S)NA3Zr4yFIQC1*p}@?CaxFJo_D(r-oJf59(bu$AlIS(d&*^9ioh zsT&3n(N`Sl|Dcz_o~rw+)KCWFU{5>ff2H=qmU24TEka@{Mzr+dXjcXu=fY zyl4fZp+3uMt-G0keSG&e`5QD=l$5h$ZtZ~vhyX{$Cvyqoezdny zj%khTNFly^)3rW=GSn5>u%TbC9I>n3uc$u-ehj&5X|^xWph>1DQvSC9vH$43i94Sj zaJCMHc%D7=QO+#qI8WAx4(0~wl`EfIpVK8cQ?hL$L*bMT>7aF4Ucj8Zu1Ol{-dkHO^NygZ){XR+j zqb+Y$@ttEuXV!H#ica?F)xqVYu3cOZzclx$AI-< zBx&lDa(PN?RO9W)t%>GAY2@2!yX@7CvD99VX~*F>AEOncnK~@bKywv{l-_xM__?9x zBgf$rKL(9c&Y_PnGUi?HA2}i1{j~G;O^pglBhBPclvKX|+J&4G;Uir-e%BQ@bA~j3 zk87Bw(X))L#^-@RJohso5Wn_B5vxK1Ei*y$G*}l~(yp`IzUrhE!3E!B4-n$UJ zh@Nt~Tsz_-Mw$C)Su9q&t=7@~;3Oi|`zKj~)|zZ$$$8pHMgO+43(;E*d1f$u(AZiK z8*zb}S*peDn7^X`9?c6;=}V@+4;$&abt_=fzYngMHJhH&S@taHwND3_^S88wa}p4I z^RP(9(o1EY+Ot%@h6qcSa-75EJ?f9~bi&K7?7a>y;@^vJ#97$@s zbol;EW6E6Mjk`5k^sJReDcXTKN2sL}^zghhR!kiB1&TY&YM z?|f4AJ85I8EkhA11Y*@}88jb0j&1*V--hN#8!%?W@EH>Uv=}g4D7b%e}^Ue!Bv_X^Loe?Bu+kw!pg( zXxb5Lo7l1G7ZxWp1nSCCRpwjUYaD)l!IZ#Xai3;3!yIf%PG5mygMa7{Cl+J$9kQ*1 z(4E0X+kNhtvI%nLe#Yw2X}h>*f?sbaygGfcv}E}UylrkJ>zq;3E2#Xh<_O(0t{cqq z_9mTUJga@}5c9f`g64OO(kRX6VBen3m`0JF&WLk~`t$W+idn6lV|B0sLdtNF9zN0R zcXz;BDUT<$SDZeHKYMXw9h=*qCaHjZ=hkpO_jC1Y@AnoX&RJWlOl)gL=If3^V2$8YADUvehQYW%}t2X+E(glq2>umr|f+zh!A^Ds z;HKDlY5%Du3fQzUmN3(84Jp`p54muPsJFA6X5&Nw-$(j4Q`t>Fi36f^*eB)LgaIeR zmBl-L-N4WwBk2-5Rn>jKQDo&W@7oROyNXtjxh22o@m6Mg{i}E*rQH%vEX~RE?i`-( zF|6F;+_-Kqv{0BGRJ5vo;7|JP^6Bm(0l=a%2jP^MGZzV{hF2v8M=THWWy{@CDxEhJI_lK9|Zk-2ap5JA-D9F+IrrS+SLa1!ne-5viEdJn}pC$;sM=1{q?xA;&9TvwWKeQfsQk~ z1KsR^o>y=W6K-8AUGi0Gu8hTeqf?#vL4E5lrcBRPqsBA&JzM=(Q-b&Q(+}e${X3OU%Ws%KkV00F-3(QjFrg z@b=^DEDeV6RW5=8UwaynguLO`JuJ_PZuwfRj16 z)j1*}Ulg|3*zmH5+qFF*i2xzB&1gZ+@^73TCl4(o4+s#GgmV=g^E`#BF!6L#34$#kI{L^?OkXpjKK!K6BwJd?Y z3hs1C`-GiDxr6LPIo>5U^Np~x4x}14+VX*D8G=fiL>)gd5r#@fK8&*t+@=z z&_7S|?-x5{EYH?oLb3#T-lTFS?Ur7sajNE$D+oe|;~rvM8x?5fTBLj6 zp}8uo@o)ER8lKHwBjTdXMB~_^1IPJZEbLF8=RZ6O>LJrn;>mu)l*_{IV3JyU`euXx zPM%HBL$CcUSMGn#h5x5iT6VUp>9%MW`}?AIL3XQi|H16pe=iEfKCjcgwhfFdasF}s U^L#4%qfvmxX)Ci*Q_s8q1CS0Pz5oCK literal 0 HcmV?d00001 From a3161c5b587decdf7ba2f540399fc30a2be807c3 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 23 Jun 2019 07:28:18 +0000 Subject: [PATCH 29/40] modifications --- .../launching_emr_cluster-1.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 5f063e15..6394e114 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,7 +3,10 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) +{{% notice note %}} +Normally our dataset on S3 would be located on the same region where we are going to run our EMR clusters. In this workshop, for educational purposes, it is fine if you are running EMR in a different region, and the Spark application will work against the dataset which is located in the N. Virginia region. +{{% /notice %}} To launch the cluster, follow these steps:\ From 89bebb821ba60ef04f5c8241862e136db0a9ff95 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 23 Jun 2019 13:39:33 +0000 Subject: [PATCH 30/40] modifications --- .../examining_cluster.md | 4 ++-- .../prerequisites_notes.md | 2 +- .../selecting_instance_types.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index d1b77a57..3f6ba041 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -6,8 +6,8 @@ weight: 95 In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. ### EMR Management Console -To get started, let's check that your EMR cluster and Spark application are running. -1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running. +To get started, let's check that your EMR cluster and Spark application are running.\ +1. In our EMR Cluster page, the status of the cluster will either be Starting (in which case you can see the status of the hardware in the Summary or Hardware tabs) or Running.\ 2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. {{% notice note %}} diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index a86c236a..d9b0c975 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -22,4 +22,4 @@ f. click **Next** and again **Next** in the next screen.\ g. Click **Create stack**.\ The stack creation should take under 2 minutes and the status of the stack will be **CREATE_COMPLETE**. -Congratulations! you completed the pre-requisites to start the workshop, you now have a VPC to run your EMR cluster in, and an S3 bucket for the Spark application code and the results. Continue to the next step to proceed in the workshop. \ No newline at end of file +Congratulations! you completed the prerequisites to start the workshop, you now have a VPC to run your EMR cluster in, and an S3 bucket for the Spark application code and the results. Continue to the next step to proceed in the workshop. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 2b734e88..2eca671e 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -36,7 +36,7 @@ Instance types with sufficient Memory and vCPUs for our executor size, as well a **Previous generation instance types:**\ - r3.xlarge and larger\ - i2.xlarge and larger\ -you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take more time to complete.\ +you will notice that these instance types have double the vCores as they do vCPU, as reflected in the EMR instance selection window - this is an EMR optimization method. Feel free to use these as well, but note that the executor calculations that we're referring to in the workshop will differ. Also, these previous generation instance types will perform slower and the application will take longer to complete.\ Also note that not all instance types exist in all regions. {{% /expand%}} From 5d173edd7893360427d993202d06fa05ae6dfed3 Mon Sep 17 00:00:00 2001 From: ranshn Date: Mon, 24 Jun 2019 17:02:57 +0000 Subject: [PATCH 31/40] adding Limits warning in launching-2 --- .../launching_emr_cluster-2.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index 0ab9130b..e4c7c332 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -25,6 +25,12 @@ Under the task node type, Click **Add / remove instance types to fleet** and sel Since our executor size is 4 vCPUs, and each instance counts as the number of its vCPUs towards the total units, let's specify **40 Spot units** in order to run 10 executors, and allow EMR to select the best instance type in the Task Instance Fleet to run the executors on. In this example, it will either start 10 * r4.xlarge / r5.xlarge / i3.xlarge **or** 5 * r5.2xlarge / r4.2xlarge in EMR Task Instance Fleet. ![FleetSelection3](/images/running-emr-spark-apps-on-spot/emrinstancefleets-task2.png) +{{% notice warning %}} +If you are using a new AWS account, or an account where Spot Instances were never launched in, your ability to launch Spot Instances will be limited. To overcome this, please make sure you launch no more than 3 instances in the Task Instance Fleet. You can do this, for example, by only providing instances that count as 8 units, and specify 24 for Spot units.\ +If your Task Instance Fleet is stuck on provisioning, try lowering the number of requested instances further. +Your Spark application should still complete successfully, but it might take longer due to having less executors in the cluster. +{{% /notice %}} + click **Next** to continue to the next steps of launching your EMR cluster. From bbdd8e9b9fe69c90bc5a0a8651d67fb5df18d80e Mon Sep 17 00:00:00 2001 From: ranshn Date: Tue, 25 Jun 2019 11:56:31 +0000 Subject: [PATCH 32/40] various mods --- .../emr_uniform_groups.md | 2 +- .../launching_emr_cluster-2.md | 8 +++++++- .../spot_savings_summary.md | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md index 55ceef30..cadb4a72 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md @@ -1,5 +1,5 @@ --- -title: "EMR Uniform Groups" +title: "EMR Uniform Instance Groups" weight: 20 --- diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md index e4c7c332..d851fd3f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-2.md @@ -5,7 +5,13 @@ weight: 70 Under "**Instance group configuration**", select Instance Fleets. Under Network, select the VPC that you deployed using the CloudFormation template earlier in the workshop, and select all subnets in the VPC. When you select multiple subnets, the EMR cluster will still be started in a single Availability Zone, but EMR Instance Fleets will make the best instance type selection based on available capacity and price across the multiple availability zones that you specified. ![FleetSelection1](/images/running-emr-spark-apps-on-spot/emrinstancefleetsnetwork.png) -Let's discuss the right setup for each of our node types:\ + + +### Setting up our EMR Master node, and Core / Task Instance Fleets +{{% notice note %}} +The workshop focuses on running Spot Instances across all the cluster node types for cost savings. If you want to dive deeper into when to use On-Demand and Spot in your EMR clusters, [click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html#emr-plan-spot-instances) +{{% /notice %}} + #### **Master node**: Unless your cluster is very short-lived and the runs are cost-driven, avoid running your Master node on a Spot Instance. We suggest this because a Spot interruption on the Master node terminates the entire cluster. \ For the purpose of this workshop, we will run the Master node on a Spot Instance as we simulate a relatively short lived job running on a transient cluster. There will not be business impact if the job fails due to a Spot interruption and later re-started.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md index 02b1bd40..a12e19d8 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/spot_savings_summary.md @@ -3,7 +3,7 @@ title: "Spot savings summary" weight: 96 --- -When our cluster has finished bootstrapping and the Spark application is running, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. +When our cluster has finished bootstrapping and the Spark application is running or has already completed, we can have a look at how much we are saving by running Spot Instances. The Spot Savings Summary feature in the EC2 Spot console provides a current snapshot of the Spot Instances in our account, and the current savings. 1. Open the AWS Management console and go to the EC2 Service in the region where you are running your EMR cluster. 2. On the left pane, click **Spot Requests**. From da0dee3330b1d834ebaf3a7ccf55627571112131 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 26 Jun 2019 09:22:02 +0000 Subject: [PATCH 33/40] various mods --- .../_index.md | 18 +++++++----------- .../emr_instance_fleets.md | 9 ++++++--- .../emr_uniform_groups.md | 16 ---------------- .../launching_emr_cluster-1.files/script.py | 6 ++++++ .../launching_emr_cluster-1.md | 9 ++++++--- .../prerequisites_notes.md | 3 ++- .../right_sizing_executors.md | 4 ++-- .../selecting_instance_types.md | 2 +- 8 files changed, 30 insertions(+), 37 deletions(-) delete mode 100644 content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index 86cc426a..fc7a5ad3 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -9,18 +9,12 @@ pre: "⁃ " ## Overview -In this workshop you will assume the role of a data engineer, tasked with building a platform that will allow your organization to run data processing jobs, specifically Apache Spark applications. +Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\ -The requirements for the platform are: +The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $1.\ +The **learning objective** for the workshop is to become familiar with the best practices and tooling that are available to you for cost optimizing your EMR clusters running Spark applications, using Spot Instances. -1. Use a managed service - in order to avoid the heavy lifting of installing, maintaining and upgrading compute clusters that run Apache Hadoop framework software, mainly Spark. -2. Be secure - allow network level isolation and encryption at rest and in transit. -3. Be cost optimized - use Amazon EC2 Spot Instances, as well as easily run transient clusters (that will be spun up just to run a job and then spun down) where possible in order to cost optimize. -4. Decouple compute and storage - in order to allow to elastically scale your processing power independently from having to provision more storage for your clusters. -5. Be self-healing in order to decrease operations overhead - if a compute node fails, the cluster will automatically replace it and continue running the job. - - -## The decision is simple - ***Amazon EMR*** fulfills all the requirements. +## Recap - Amazon EMR and EC2 Spot Instances * [Amazon EMR] (https://aws.amazon.com/emr/) provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as [Apache Spark] (https://aws.amazon.com/emr/details/spark/), [HBase] (https://aws.amazon.com/emr/details/hbase/), [Presto] (https://aws.amazon.com/emr/details/presto/), and [Flink] (https://aws.amazon.com/blogs/big-data/use-apache-flink-on-amazon-emr/) in EMR, and interact with data in other AWS data stores such as Amazon S3 and Amazon DynamoDB. EMR Notebooks, based on the popular Jupyter Notebook, provide a development and collaboration environment for ad hoc querying and exploratory analysis. EMR securely and reliably handles a broad set of big data use cases, including log analysis, web indexing, data transformations (ETL), machine learning, financial analysis, scientific simulation, and bioinformatics. @@ -28,4 +22,6 @@ The requirements for the platform are: * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. ### About Spot Instances in Analytics workloads -The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as get our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back. It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in details during this workshop. \ No newline at end of file +The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as achieve our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back.\ +It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in detail during this workshop. + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md index 76e64bb5..0104f3a9 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/emr_instance_fleets.md @@ -3,13 +3,16 @@ title: "EMR Instance Fleets" weight: 30 --- -With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones. +When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. + +With EMR instance fleets, you specify target capacities for On-Demand Instances and Spot Instances within each fleet (Master, Core, Task). When the cluster launches, Amazon EMR provisions instances until the targets are fulfilled. You can specify up to five EC2 instance types per fleet for Amazon EMR to use when fulfilling the targets. You can also select multiple subnets for different Availability Zones.\ {{% notice info %}} -[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets. +[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) to learn more about EMR Instance Fleets in the official documentation. {{% /notice %}} **When Amazon EMR launches the cluster, it looks across those subnets to find the instances and purchasing options you specify, and will select the Spot Instances with the lowest chance of getting interrupted, for the lowest cost.** -While a cluster is running, if Amazon EC2 reclaims a Spot Instance or if an instance fails, Amazon EMR tries to replace the instance with any of the instance types that you specify in your fleet. This makes it easier to regain capacity in case some of the instances get interrupted by EC2 when it needs the Spot capacity back. \ No newline at end of file +While a cluster is running, if Amazon EC2 reclaims a Spot Instance or if an instance fails, Amazon EMR tries to replace the instance with any of the instance types that you specify in your fleet. This makes it easier to regain capacity in case some of the instances get interrupted by EC2 when it needs the Spot capacity back.\ +These options do not exist within the default EMR configuration option "Uniform Instance Groups", hence we will be using EMR Instance Fleets only. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md b/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md deleted file mode 100644 index cadb4a72..00000000 --- a/content/running_spark_apps_with_emr_on_spot_instances/emr_uniform_groups.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "EMR Uniform Instance Groups" -weight: 20 ---- - -When using the EMR console to create a cluster via the quick settings, default advanced settings, or with the AWS CLI - Amazon EMR will provision your EMR cluster with a configuration option called "Uniform Instance Groups". - -With the Uniform Groups configuration option, you select the Availability Zone in which you want to launch your EMR cluster, and one instance type per group (Master, Core, and optional multiple Task groups). - -{{% notice info %}} -[Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to learn more about Master, Core and Task node types in EMR. -{{% /notice %}} - -When adopting Spot Instances into your workload, it is recommended to be flexible around how to launch your workload in terms of Availability Zone and Instance Types. This is in order to be able to achieve the required scale from multiple Spot capacity pools (a combination of EC2 instance type in an availability zone) or one capacity pool which has sufficient capacity, as well as decrease the impact on your workload in case some of the Spot capacity is interrupted with a 2-minute notice when EC2 needs the capacity back, and allow EMR to replenish the capacity with a different instance type. - -We will not use EMR Uniform Groups configuration option in this workshop. Instead, we will focus on the more robust and Spot friendly configuration option - **EMR Instance Fleets** - continue the workshop to learn more and use this configuration option. \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py new file mode 100644 index 00000000..485a7c97 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.files/script.py @@ -0,0 +1,6 @@ +import sys +from pyspark.sql import SparkSession +spark = SparkSession.builder.appName('Amazon reviews word count').getOrCreate() +df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") +df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) +exit() \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md index 6394e114..4e68c2cc 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/launching_emr_cluster-1.md @@ -3,7 +3,7 @@ title: "Launch a cluster - Step 1" weight: 60 --- -In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) +In this step we'll launch our first cluster. This will be a transient cluster that will be shut down after it finishes running the application we submit to it at cluster creation time, and will run solely on Spot Instances. The application is a simple wordcount that will run against a public data set of Amazon product reviews, located in an Amazon S3 bucket in the N. Virginia region. If you want to know more about the Amazon Customer Reviews Dataset, [click here] (https://s3.amazonaws.com/amazon-reviews-pds/readme.html) {{% notice note %}} Normally our dataset on S3 would be located on the same region where we are going to run our EMR clusters. In this workshop, for educational purposes, it is fine if you are running EMR in a different region, and the Spark application will work against the dataset which is located in the N. Virginia region. {{% /notice %}} @@ -20,7 +20,7 @@ To launch the cluster, follow these steps:\ ``` --executor-memory 18G --executor-cores 4 ``` -* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp \ s3://\** +* **Application location**: here we will configure the location of our Spark application. Save the following python code to a file (or download it from the Attachment box) and upload it to your S3 bucket using the AWS console or AWS CLI: **aws s3 cp script.py s3://\** ```python import sys @@ -30,7 +30,10 @@ df = spark.read.parquet("s3://amazon-reviews-pds/parquet/") df.selectExpr("explode(split(lower(review_body), ' ')) as words").groupBy("words").count().write.mode("overwrite").parquet(sys.argv[1]) exit() ``` -Then add the location of the file under the **Application location** field, i.e: s3://\/\\ +{{%attachments style="orange" /%}} + + +Then add the location of the file under the **Application location** field, i.e: s3://\/script.py * **Arguments**: Here we will configure the location of where Spark will write the results of the job. Enter: s3://\/results/\ * **Action on failure**: Leave this on *Continue* and click **Add** to save the step. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md index d9b0c975..a0e922cd 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/prerequisites_notes.md @@ -11,7 +11,8 @@ weight: 10 #### Preparation steps:\ -1. Create an S3 bucket for your Spark application code (which will be provided later) and the EMR application's results. Using the AWS CLI, run: **aws s3 mb s3://\** or create a new bucket using the AWS Management Console. +1. Create an S3 bucket for your Spark application code (which will be provided later) and the Spark application's results.\ +Using the AWS CLI, run: **aws s3 mb s3://\** or create a new bucket using the AWS Management Console. 2. Deploy a new VPC that will be used to run your EMR cluster in the workshop.\ a. Open the ["Modular and Scalable VPC Architecture Quick stage page"] (https://aws.amazon.com/quickstart/architecture/vpc/) and go to the "How to deploy" tab, Click the ["Launch the Quick Start"] (https://fwd.aws/mm853) link.\ b. Select your desired region to run the workshop from the top right corner of the AWS Management Console and click **Next**.\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md index 9465f7fa..34a8497c 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.md @@ -3,13 +3,13 @@ title: "Right sizing Spark executors" weight: 40 --- -Building towards the running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important step that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. +Building towards running the first Spark application on Amazon EMR Instance Fleets, let's dive deeper into the most important best practice that will allow us to be **flexible** around our EC2 Instance type selection - **right sizing our Spark executors**. {{% notice note %}} **Remember!** you might be able to achieve greater utilization and performance optimization when using a single EC2 instance type, but when adopting Spot Instances, the idea is to be as flexible as possible in order to achieve and keep the desired scale to run your Spark applications. {{% /notice %}} -Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of instances. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. +Today, the developers in your organization run the job on a non-managed Spark cluster using “**spark-submit —executor-memory=72G —executor-cores=16**". This confines the platform to EC2 instance types with a lot of memory and cores to accommodate the size of the executor, and prevents the cluster from running on a diversified set of EC2 instance types. If the Spot interruption rate for the high memory instances will increase, then you will have problems getting and keeping capacity for the clusters, hence we need to right size the executors to fit on smaller instance types and allow for a larger instance type selection. In some cases, smaller instance types will have lower Spot interruption rates. We will find out how to see Spot Interruption rates for the different instance types in the next section. If we keep approximately the same vCPU:Mem ratio (1:4.5) for our job and avoid going over the recommended Java memory configuration (20-30GB), then we'll conclude that we can optimize our executor configuration by using "--executor-memory=24GB --executor cores=4". However, there are some more limitations in place. diff --git a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md index 2eca671e..a7b02b64 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/selecting_instance_types.md @@ -6,7 +6,7 @@ weight: 50 Let's use our newly acquired knowledge around Spark executor sizing in order to select the EC2 Instance Types that will be used in our EMR cluster.\ EMR clusters run Master, Core and Task node types. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html) to read more about the different node types. -We determined that in order to maximize usage of R4 instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, +We determined that in order to be flexible and allow running on multiple instance types, we will submit our Spark application with **"–executor-memory=18GB –executor cores=4"**, We can use the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) page to find the relevant instance types with sufficient number of vCPUs and RAM, and use this opportunity to also select instance types with low interruption rates. \ For example: r5.2xlarge has 8 vCPUs and 64 GB of RAM, so EMR will automatically run 2 executors that will consume 36 GB of RAM and still leave free RAM for the operating system and other processes.\ From 86a510fb43c518f33101c6c7c3c8600f20fe7bb4 Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 26 Jun 2019 09:36:37 +0000 Subject: [PATCH 34/40] various mods --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index fc7a5ad3..fe57febf 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -21,7 +21,7 @@ The **learning objective** for the workshop is to become familiar with the best * [Amazon EC2 Spot Instances] (https://aws.amazon.com/ec2/spot/) offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads. -### About Spot Instances in Analytics workloads +## About Spot Instances in Analytics workloads The most important best practice when using Spot Instances is to be flexible with the EC2 instance types that our application can run on, in order to be able to access many spare capacity pools (a combination of EC2 instance type and an Availability Zone), as well as achieve our desired capacity from a different instance type in case some of our Spot capacity in the EMR cluster is interrupted, when EC2 needs the spare capacity back.\ It's possible to run Spark applications in a single cluster that is running on multiple different instance types, we'll just need to right-size our executors and use the EMR Instance Fleets configuration option in order to meet the Spot diversification best practice. We'll look into that in detail during this workshop. From 63bc1611f20c9c745b814c9b6481caa3ce6f4e08 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 09:24:37 +0000 Subject: [PATCH 35/40] various mods --- .../examining_cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index 3f6ba041..ec99296f 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -23,7 +23,7 @@ Some notable metrics: * AppsRunning - you should see 1 since we only submitted one step to the cluster.\ * ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ -* MemoryAllocatedMB * MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ +* MemoryAllocatedMB & MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ ### Using Ganglia and YARN ResourceManager The recommended approach to connect to the web interfaces running on our EMR cluster is to use SSH tunneling. [Click here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) to learn more about connecting to EMR interfaces.\ From f075bc418bc094fc6a27666062031e331c958c45 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:09:14 +0000 Subject: [PATCH 36/40] various mods --- .../simulating_recovery.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md diff --git a/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md b/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md new file mode 100644 index 00000000..d6b58521 --- /dev/null +++ b/content/running_spark_apps_with_emr_on_spot_instances/simulating_recovery.md @@ -0,0 +1,30 @@ +--- +title: "(Optional) simulating recovery" +weight: 149 +--- + +EMR replenishes the target capacity if some EC2 Instances failed, were terminated/stopped, or received an EC2 Spot Interruption. + +In this optional step, you will re-run the cluster and the Spark application, and terminate some of the Task Fleet instances in order to observe the recovery capabilities of EMR and Spark, and check that the application still completes successfully. Since it is not possible to simulate an EC2 Spot Interruption in an EMR cluster, we will have to manually terminate EC2 instances to receive a similar effect. + +{{% notice note %}} +This is an optional step that will take approximately 20 minutes more than the original running time of the workshop. Feel free to skip this step and go directly to the **Conclusions and cleanup** step. +{{% /notice %}} + +#### Step objectives:\ +1. Observe that EMR replenishes the target capacity if some EC2 Instances failed, were terminated/stopped, or received an EC2 Spot Interruption. +2. Observe that the Spark application running in your EMR cluster still completes successfully, despite losing executors due to instance terminations. + +#### Re-launch your cluster and application +1. In the EMR console, select the cluster that you launched in this workshop, and click the **Clone** button. +2. In the popup dialog **"Would you like to include steps"** select **Yes** and click **Clone**. +3. EMR console duplicated all the cluster settings for you - click the **Create cluster** button. +4. Refresh the Summary tab in the EMR console until the status of the cluster is **Running step** and the Master, Core and Task fleets are all in the **Running** state. + +#### Manually terminate some of the EMR Task Fleet nodes +1. Go to the EC2 console, and identify the instances in your Task Fleet. You can do so by using the following filter: **Key=aws:elasticmapreduce:instance-group-role & Value=TASK**. If you have other EMR clusters running in the account/region, make sure you identify your own cluster by further filtering according to its Name tag. +2. Randomly select half of the instances that were filtered, and click Actions -> Instance State -> Terminate -> **Yes, Terminate** + +#### Verify that EMR replenished the capacity, and that the step completed successfully +1. Within 2-3 minutes, refresh the EC2 console as well as the Task Fleet in the EMR console under the Hardware tab, and you should see new Task Fleet instances created by EMR to replenish the capacity, after you terminated the previous instances. +2. In the EMR console, go to the Steps tab. refresh the tab until your application has reached the **Completed status**. Because some instances were terminated mid-run, the Step will still complete, but will take longer than you previously observed in the workshop, because Spark had to repeat some of the work. \ No newline at end of file From 3143864c07b85538e375ebe6804f4b85a0c9e301 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:14:00 +0000 Subject: [PATCH 37/40] various mods --- .../conclusions_and_cleanup.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md index a4111f7e..2550ff92 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md @@ -11,7 +11,8 @@ weight: 150 2. Delete the VPC you deployed via CloudFormation, by going to the CloudFormation service in the AWS Management Console, selecting the VPC stack (default name is Quick-Start-VPC) and click the Delete option. Make sure that the deletion has completed successfully (this should take around 1 minute), the status of the stack will be DELETE_COMPLETE (the stack will move to the Deleted list of stacks). 3. Delete your S3 bucket from the AWS Management Console - choose the bucket from the list of buckets and hit the Delete button. This approach will also empty the bucket and delete all existing objects in the bucket. 4. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**. + #### Thank you -We hope this workshop was educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR in order to optimize your costs.\ +We hope you found this workshop educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR, in order to optimize your costs.\ If you have any feedback or questions, click the "**Feedback / Questions?**" link in the left pane to reach out to the authors of the workshop. \ No newline at end of file From 0bf4dec1ee9f0593de196e1e733cb518223c6e7b Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 13:16:43 +0000 Subject: [PATCH 38/40] various mods --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index fe57febf..c9c11658 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -11,7 +11,7 @@ pre: "⁃ " Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\ -The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $1.\ +The **estimated time** for completing the workshop is 60-90 minutes and the **estimated cost** for running the workshop's resources in your AWS account is less than $2.\ The **learning objective** for the workshop is to become familiar with the best practices and tooling that are available to you for cost optimizing your EMR clusters running Spark applications, using Spot Instances. ## Recap - Amazon EMR and EC2 Spot Instances From 2f2191739680e02b0624314cb0e6ccf122848f10 Mon Sep 17 00:00:00 2001 From: ranshn Date: Sun, 30 Jun 2019 16:53:33 +0000 Subject: [PATCH 39/40] various mods --- .../{visualizing_costs.md => analyzing_costs.md} | 11 ++++++----- .../examining_cluster.md | 14 +++++++------- .../fleet_config_options.md | 10 +++++----- .../tracking_spot_interruptions.md | 6 +++--- 4 files changed, 21 insertions(+), 20 deletions(-) rename content/running_spark_apps_with_emr_on_spot_instances/{visualizing_costs.md => analyzing_costs.md} (87%) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md b/content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md similarity index 87% rename from content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md rename to content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md index 9c37530e..648ccfd7 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/visualizing_costs.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md @@ -1,5 +1,5 @@ --- -title: "Visualizing costs" +title: "Analyzing costs" weight: 145 --- @@ -22,15 +22,16 @@ If the Name tag Key was not enabled as a Cost Allocation Tag, you will not be ab Let's use Cost Explorer to analyze the costs of running our EMR application.\ 1. Navigate to Cost Explorer by opening the AWS Management Console -> Click your username in the top right corner -> click **My Billing Dashboard** -> click **Cost Explorer in the left pane**. or [click here] (https://console.aws.amazon.com/billing/home#/costexplorer) for a direct link.\ 2. We know that we gave our EMR cluster a unique Name tag, so let's filter according to it. In the right pane, click Tags -> Name -> enter "**EMRTransientCluster1**"\ -3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end. -4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia). +3. Instead of the default 45 days view, let's narrow down the time span to just the day when we ran the cluster. In the data selection dropdown, mark that day as start and end.\ +4. You are now looking at the total cost to run the cluster (**$0.30**), including: EMR, EC2, EBS, and possible AWS Cross-Region data transfer costs, depending on where you ran your cluster relative to where the S3 dataset is located (in N. Virginia).\ 5. Group by **Usage Type** to get a breakdown of the costs ![costexplorer](/images/running-emr-spark-apps-on-spot/costexplorer1.png) -* EU-SpotUsage:r5.xlarge: This was the instance that I ran in the EMR Task Instance fleet and accured the biggest cost ($0.17)\ +* EU-SpotUsage:r5.xlarge: This was the instance type that ran in the EMR Task Instance fleet and accrued the largest cost, since EMR launched 10 instances ($0.17)\ * EU-BoxUsage:r5.xlarge: The EMR costs. [Click here] (https://aws.amazon.com/emr/pricing/) to learn more about EMR pricing. ($0.06)\ * EU-EBS:VolumeUsage.gp2: EBS volumes that were attached to my EC2 Instances in the cluster - these got tagged automatically. ($0.03)\ * EU-SpotUsage:r5a.xlarge & EU-SpotUsage:m4.xlarge: EC2 Spot price for the other instances in my cluster (Master and Core) ($0.02 combined)\ -If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you Filter by Service=EMR and Group by Usage Type? \ No newline at end of file +If you have access to Cost Explorer, have a look around and see what you can find by slicing and dicing with filtering and grouping. For example, what happens if you filter by **Purchase Option = Spot** & **Group by = Instance Type**? + diff --git a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md index ec99296f..d84d3ad1 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md @@ -3,7 +3,7 @@ title: "Examining the cluster" weight: 95 --- -In this section we will look at the utilization of our instances while the application is running, to examine how many Spark executors we are running and what is the instances utilization. +In this section we will look at the utilization of our EC2 Spot Instances while the application is running, and examine how many Spark executors are running. ### EMR Management Console To get started, let's check that your EMR cluster and Spark application are running.\ @@ -11,7 +11,7 @@ To get started, let's check that your EMR cluster and Spark application are runn 2. Move to the Steps tab, and your Spark application will either be Pending (for the cluster to start) or Running. {{% notice note %}} -In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very intensive and is just used for demo purposes. +In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very resource intensive and is just used for demo purposes. {{% /notice %}} ### Using CloudWatch Metrics @@ -22,7 +22,7 @@ EMR emits several useful metrics to CloudWatch metrics. You can use the AWS Mana Some notable metrics: * AppsRunning - you should see 1 since we only submitted one step to the cluster.\ -* ContainerAllocated - this represents the number of container Spark executors that are running on your cluster, on the Core and Task Instance Fleets.\ +* ContainerAllocated - this represents the number of containers that are running on your cluster, on the Core and Task Instance Fleets. These would the be Spark executors and the Spark Driver.\ * MemoryAllocatedMB & MemoryAvailableMB - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.\ ### Using Ganglia and YARN ResourceManager @@ -34,15 +34,15 @@ Normally you would not run EMR in a public subnet and open TCP access to the mas {{% /notice %}} To allow access to your IP address to reach the EMR web interfaces via EC2 Security Groups:\ -1. In your EMR cluster page, in the AWS Management Console, go to the Summary tab\ +1. In your EMR cluster page, in the AWS Management Console, go to the **Summary** tab\ 2. Click on the ID of the security under **Security groups for Master**\ 3. Check the Security Group with the name **ElasticMapReduce-master**\ -4. In the lower pane, click the **Inbound tab** and click the Edit button\ +4. In the lower pane, click the **Inbound tab** and click the **Edit**\ 5. Click **Add Rule**. Under Type, select **All Traffic**, under Source, select **My IP**\ -6. Click **Save**. +6. Click **Save** {{% notice note %}} -While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which is not allowed for outbound traffic on every Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) +While the Ganglia web interface uses TCP port 80, the YARN ResourceManager web interface uses TCP port 8088 which might not allowed for outbound traffic on your Internet connection. If you are using a network connection that blocks TCP 8088 (or in other words, doesn't allow non-well known ports) then you will not be able to reach the YARN ResourceManager web interface. You can either skip that part of the workshop, or consider using the more complex method of SSH tunneling described [here] (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html) {{% /notice %}} Go back to the Summary tab in your EMR cluster page, and you will see links to tools in the **Connections** section (you might need to refresh the page).\ diff --git a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md index 065696a8..b66b4927 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/fleet_config_options.md @@ -3,19 +3,19 @@ title: "Fleet configuration options" weight: 85 --- -While our cluster is starting (7-8 minutes) and the job is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. +While our cluster is starting (7-8 minutes) and the step is running (4-8 minutes depending on the instance types that were selected) let's take the time to look at some of the EMR Instance Fleets configurations we didn't dive into when starting the cluster. ![fleetconfigs](/images/running-emr-spark-apps-on-spot/emrinstancefleets-core1.png) #### Maximum Spot price -Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. +Since Amazon EC2 Spot Instances [changed the pricing model and bidding is no longer required] (https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/), we have an optional "Max-price" field for our Spot requests, which would limit how much we're willing to pay for the instance. It is recommended to leave this value at 100% of the On-Demand price, in order to avoid limiting our instance diversification. We are going to pay the Spot market price regardless of the Maximum price that we can specify, and setting a higher max price does not increase the chance of getting Spot capacity nor does it decrease the chance of getting your Spot Instances interrupted when EC2 needs the capacity back. You can see the current Spot price in the AWS Management Console under EC2 -> Spot Requests -> **Pricing History**. #### Each instance counts as X units -This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of vCPUs that the instance type has - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 80, then 20 * r4.xlarge instances will be launched by EMR for our Core Instance Fleet. -If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the Core fleet. Since our executor size is 18 GB, one executor will run on this instance type. +This configuration allows us give each instance type in our diversified fleet a weight that will count towards our Total units. By default, this weight is configured as the number of YARN VCores that the instance type has by default (this would typically equate to the number of EC2 vCPUs) - this way it's easy to set the Total units to the number of vCPUs we want our cluster to run with, and EMR will select the best instances while taking into account the required number of instances to run. For example, if r4.xlarge is the instance type that EMR found to be the least likely to be interrupted and has the lowest price out of our selection, its weight is 4 and our total units (only Spot) is 40, then 10 * r4.xlarge instances will be launched by EMR in the fleet. +If my Spark application is memory driven, I can set the total units to the total amount of memory I want my cluster to run with, and change the "Each instance counts as" field to the total memory of the instance, leaving aside some memory for the operating system and other processes. For example, for the r4.xlarge I can set its weight to 25. If I then set up the Total units to 500 then EMR will bring up 20 * r4.xlrage instances in the fleet. Since our executor size is 18 GB, one executor will run on this instance type. #### Defined duration This option will allow you run your EMR Instance Fleet on Spot Blocks, which are uninterrupted Spot Instances, available for 1-6 hours, at a lower discount compared to Spot Instances. #### Provisioning timeout -You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the Fleet when looking to use Spot Instances, before you look into failing over to On-Demand. \ No newline at end of file +You can determine that after a set amount of minutes, if EMR is unable to provision your selected Spot Instances due to lack of capacity, it will either start On-Demand instances instead, or terminate the cluster. This can be determined according to the business definition of the cluster or Spark application - if it is SLA bound and should complete even at On-Demand price, then the "Switch to On-Demand" option might be suitable. However, make sure you diversify the instance types in the fleet when looking to use Spot Instances, before you look into failing over to On-Demand. Also, try to select instance types with lower interruption rates according to the [Spot Instance Advisor] (https://aws.amazon.com/ec2/spot/instance-advisor/) \ No newline at end of file diff --git a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md index b01eaa7e..44c16096 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/tracking_spot_interruptions.md @@ -6,7 +6,7 @@ weight: 100 Now we're in the process of getting started with adopting Spot Instances for our EMR clusters. We're still not sure that our jobs are fully resilient and what would actually happen if some of the EC2 Spot Instances in our EMR clusters get interrupted, when EC2 needs the capacity back for On-Demand. {{% notice note %}} -In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful. +In most cases, when running fault-tolerant workloads, we don't really need to track the Spot interruptions as our applications should be built to handle them gracefully without any impact to performance or availability, but when we get started with EMR jobs this could be useful, as our organization can use these to correlate to possible EMR job failures or prolonged execution times, in case Spot Instances were interrupted during Spark run time. {{% /notice %}} @@ -32,7 +32,7 @@ aws sns subscribe --topic-arn --protocol email --notification-endpoi 1. On the right side of the console, click **Add Target**, scroll down and select **SNS topic** -> select your topic name, Your result should look like this: ![tags](/images/running-emr-spark-apps-on-spot/cloudwatcheventsrule.png) 1. Click **Configure Details** in the bottom right corner. -1. Give a name to your CloudWatch Events rule and click **Create rule**. +1. Provide a name to your CloudWatch Events rule and click **Create rule**. #### Verifying that the notification works @@ -50,4 +50,4 @@ The only way to simulate a Spot Interruption Notification is to use Spot Fleet. Go ahead and terminate the fleet request itself by checking the fleet, click actions -> **Cancel Spot request** -> **Confirm**. - +From now on, any EC2 Spot interruption in the account/region that you set this up in will alert you via email. Disable or delete the CloudWatch Event rule if you are not interested in the notifications. From d05eae907d28e06dd2237130969bde1db7cb36ef Mon Sep 17 00:00:00 2001 From: ranshn Date: Wed, 3 Jul 2019 11:29:21 +0000 Subject: [PATCH 40/40] final commit before merging --- content/running_spark_apps_with_emr_on_spot_instances/_index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/running_spark_apps_with_emr_on_spot_instances/_index.md b/content/running_spark_apps_with_emr_on_spot_instances/_index.md index c9c11658..23f8c341 100644 --- a/content/running_spark_apps_with_emr_on_spot_instances/_index.md +++ b/content/running_spark_apps_with_emr_on_spot_instances/_index.md @@ -5,8 +5,6 @@ weight: 60 pre: "⁃ " --- -## This workshop is still under construction. ping ranshein@amazon.com if you have any concerns. - ## Overview Welcome! In this workshop you will assume the role of a data engineer, tasked with cost optimizing the organization's costs for running Spark applications, using Amazon EMR and EC2 Spot Instances.\