From 65d5a59c992275de050c99f5c4666ddb075a521f Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Fri, 30 Sep 2022 21:04:37 +0000 Subject: [PATCH 1/9] update mnmg notebook and test file --- ...nguage-modeling-multi-gpu-multi-node.ipynb | 190 +++--- .../scripts/launch_sm_training_compiler.py | 9 - .../scripts/requirements.txt | 2 + .../scripts/run_clm_smddp.py | 582 +++++++++++++++++ .../scripts/run_mlm_smddp.py | 607 ++++++++++++++++++ 5 files changed, 1272 insertions(+), 118 deletions(-) delete mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/launch_sm_training_compiler.py create mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/requirements.txt create mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py create mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index ac6c157bfe..d3f93409d4 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "perfect-paraguay", + "id": "3e95cc90", "metadata": {}, "source": [ "# Compile and Train the GPT2 Model using the Transformers Trainer API with the SST2 Dataset for Multi-Node Multi-GPU Training" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "facial-classification", + "id": "444fe156", "metadata": {}, "source": [ "1. [Introduction](#Introduction) \n", @@ -25,7 +25,7 @@ }, { "cell_type": "markdown", - "id": "outer-citizenship", + "id": "4bf9ceb2", "metadata": {}, "source": [ "## SageMaker Training Compiler Overview\n", @@ -38,16 +38,16 @@ "\n", "## Introduction\n", "\n", - "In this demo, you'll use Hugging Face's `transformers` and `datasets` libraries with Amazon SageMaker Training Compiler to train the `gpt-2` model on the `Stanford Sentiment Treebank v2 (SST2)` dataset. To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on. \n", + "In this demo, you'll use Hugging Face's transformers and datasets libraries with Amazon SageMaker Training Compiler to train the gpt-2 model on the Stanford Sentiment Treebank v2 (SST2) dataset. To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on. \n", "\n", - "**NOTE:** You can run this demo in SageMaker Studio, SageMaker notebook instances, or your local machine with AWS CLI set up. If using SageMaker Studio or SageMaker notebook instances, make sure you choose one of the PyTorch-based kernels, `Python 3 (PyTorch x.y Python 3.x CPU Optimized)` or `conda_pytorch_p36` respectively.\n", + "**NOTE:** You can run this demo in SageMaker Studio, SageMaker notebook instances, or your local machine with AWS CLI set up. If using SageMaker Studio or SageMaker notebook instances, make sure you choose one of the PyTorch-based kernels, Python 3 (PyTorch x.y Python 3.x CPU Optimized) or conda_pytorch_p36 respectively.\n", "\n", - "**NOTE:** This notebook uses four `ml.p3.16xlarge` instances that have multiple GPUs. If you don't have enough quota, see [Request a service quota increase for SageMaker resources](https://docs.aws.amazon.com/sagemaker/latest/dg/regions-quotas.html#service-limit-increase-request-procedure). " + "**NOTE:** This notebook uses four ml.p4d.24xlarge instances that have multiple GPUs. If you don't have enough quota, see [Request a service quota increase for SageMaker resources](https://docs.aws.amazon.com/sagemaker/latest/dg/regions-quotas.html#service-limit-increase-request-procedure). " ] }, { "cell_type": "markdown", - "id": "authentic-phoenix", + "id": "324b417c", "metadata": {}, "source": [ "## Development Environment " @@ -55,38 +55,38 @@ }, { "cell_type": "markdown", - "id": "ethical-dubai", + "id": "f6ef966d", "metadata": {}, "source": [ "### Installation\n", "\n", - "This example notebook requires the **SageMaker Python SDK v2.70.0** and **transformers v4.11.0**." + "This example notebook requires the **SageMaker Python SDK v2.108.0** and **transformers v4.21**." ] }, { "cell_type": "code", "execution_count": null, - "id": "caring-equivalent", + "id": "a646c234", "metadata": {}, "outputs": [], "source": [ - "!pip install -U sagemaker==2.70.0" + "!pip install \"sagemaker>=2.108.0\" botocore boto3 awscli s3fs typing-extensions \"torch==1.11.0\" pandas numpy --upgrade" ] }, { "cell_type": "code", "execution_count": null, - "id": "dutch-october", + "id": "91ecb4bb", "metadata": {}, "outputs": [], "source": [ - "!pip install transformers==4.11.0" + "!pip install \"transformers==4.21\" datasets --upgrade" ] }, { "cell_type": "code", "execution_count": null, - "id": "proprietary-breath", + "id": "56fb1066", "metadata": {}, "outputs": [], "source": [ @@ -102,10 +102,10 @@ }, { "cell_type": "markdown", - "id": "orange-chess", + "id": "5a03c1ef", "metadata": {}, "source": [ - "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for `datasets` library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", + "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for datasets library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", "\n", "```python\n", "%%capture\n", @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "upper-plumbing", + "id": "814b7ca3", "metadata": {}, "source": [ "### SageMaker environment " @@ -127,7 +127,7 @@ { "cell_type": "code", "execution_count": null, - "id": "front-fetish", + "id": "ad39a3f6", "metadata": {}, "outputs": [], "source": [ @@ -152,21 +152,21 @@ }, { "cell_type": "markdown", - "id": "respiratory-morris", + "id": "55e4e7eb", "metadata": {}, "source": [ "## SageMaker Training Job\n", "\n", - "To create a SageMaker training job, we use a `HuggingFace` estimator. Using the estimator, you can define which training script should SageMaker use through `entry_point`, which `instance_type` to use for training, which `hyperparameters` to pass, and so on.\n", + "To create a SageMaker training job, we use a HuggingFace/PyTorch estimator. Using the estimator, you can define which training script should SageMaker use through entry_point, which instance_type to use for training, which hyperparameters to pass, and so on.\n", "\n", - "When a SageMaker training job starts, SageMaker takes care of starting and managing all the required machine learning instances, picks up the `HuggingFace` Deep Learning Container, uploads your training script, and downloads the data from `sagemaker_session_bucket` into the container at `/opt/ml/input/data`.\n", + "When a SageMaker training job starts, SageMaker takes care of starting and managing all the required machine learning instances, picks up the HuggingFace Deep Learning Container, uploads your training script, and downloads the data from sagemaker_session_bucket into the container at /opt/ml/input/data.\n", "\n", - "In the following section, you learn how to set up two versions of the SageMaker `HuggingFace` estimator, a native one without the compiler and an optimized one with the compiler." + "In the following section, you learn how to set up two versions of the SageMaker HuggingFace/PyTorch estimator, a native one without the compiler and an optimized one with the compiler." ] }, { "cell_type": "markdown", - "id": "retired-albert", + "id": "5cdf928c", "metadata": {}, "source": [ "### Training Setup" @@ -175,7 +175,7 @@ { "cell_type": "code", "execution_count": null, - "id": "domestic-richardson", + "id": "6a87f6e7", "metadata": {}, "outputs": [], "source": [ @@ -196,35 +196,35 @@ "\n", "# SageMaker Training Compiler currently only supports training on GPU\n", "# Select Instance type for training\n", - "INSTANCE_TYPE = \"ml.p3.16xlarge\" # ml.p3.8xlarge is easily available. However, ml.p3.16xlarge provides better performance.\n", + "INSTANCE_TYPE = \"ml.p4d.24xlarge\" # ml.p3.8xlarge is easily available. However, ml.p3.16xlarge provides better performance.\n", "NUM_INSTANCES = 2\n", "num_gpus_per_instance = 8" ] }, { "cell_type": "markdown", - "id": "final-newark", + "id": "630bed45", "metadata": {}, "source": [ - "### Training with Native PyTorch" + "### Training with SageMaker Distributed Data Parallel " ] }, { "cell_type": "markdown", - "id": "representative-emerald", + "id": "0af1f969", "metadata": {}, "source": [ - "The batch size below is the maximum batch we could fit into the memory of an `ml.p3.16xlarge` instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use AMP for faster training." + "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use AMP for faster training." ] }, { "cell_type": "code", "execution_count": null, - "id": "statutory-electricity", + "id": "8f8df334", "metadata": {}, "outputs": [], "source": [ - "from sagemaker.huggingface import HuggingFace\n", + "from sagemaker.pytorch import PyTorch\n", "\n", "# hyperparameters are passed to the training entrypoint as arguments\n", "hyperparameters = {\n", @@ -235,40 +235,31 @@ " \"do_train\": True,\n", " \"do_eval\": True,\n", " \"fp16\": True,\n", - " \"per_device_train_batch_size\": 10,\n", - " \"per_device_eval_batch_size\": 16,\n", + " \"per_device_train_batch_size\": 32,\n", " \"overwrite_output_dir\": True,\n", " \"num_train_epochs\": EPOCHS,\n", " \"output_dir\": \"/opt/ml/model\",\n", " SEQ_LEN_ARG: 512,\n", " \"logging_strategy\": \"epoch\",\n", " \"save_strategy\": \"no\",\n", + " \"preprocessing_num_workers\": 12,\n", "}\n", "\n", - "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", - "hyperparameters[\"learning_rate\"] = (\n", - " float(\"5e-5\")\n", - " / 32\n", - " * hyperparameters[\"per_device_train_batch_size\"]\n", - " * num_gpus_per_instance\n", - " * NUM_INSTANCES\n", - ")\n", - "\n", "# configure the training job\n", - "native_estimator = HuggingFace(\n", - " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\",\n", + "native_estimator = PyTorch(\n", + " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}_smddp.py\",\n", " source_dir=\"./scripts\",\n", " instance_type=INSTANCE_TYPE,\n", " instance_count=NUM_INSTANCES,\n", " role=role,\n", - " volume_size=200,\n", - " transformers_version=\"4.11\",\n", - " pytorch_version=\"1.9\",\n", + " volume_size=512,\n", + " transformers_version=\"4.21.1\",\n", + " framework_version=\"1.11.0\",\n", " py_version=\"py38\",\n", " hyperparameters=hyperparameters,\n", " distribution={\n", " \"smdistributed\": {\"dataparallel\": {\"enabled\": True}}\n", - " }, # Use SageMaker Data Parallel to train across nodes/GPUs.\n", + " }, # Use SageMaker Data Distributed Parallel to train across nodes/GPUs.\n", " disable_profiler=True, # Disabling SageMaker Profiler to avoid overhead during benchmarking\n", " debugger_hook_config=False, # Disabling SageMaker Debugger to avoid overhead during benchmarking\n", ")\n", @@ -280,70 +271,49 @@ }, { "cell_type": "markdown", - "id": "aggressive-treaty", + "id": "98e98391", "metadata": {}, "source": [ - "### Training with Optimized PyTorch" + "### Training with SageMaker Training Compiler " ] }, { "cell_type": "markdown", - "id": "perfect-trinity", + "id": "9dbbfdd5", "metadata": {}, "source": [ "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately.\n", "\n", - "**Note:** We recommend you to turn the SageMaker Debugger's profiling and debugging tools off when you use compilation to avoid additional overheads.\n", - "\n", - "Here, instead of using the `distribution` kwarg to launch a multi node training job, we use a wrapper script to setup an inter-node communication using `torch_xla.distributed.sm_dist`, which has been optimized to work with SageMaker Training Compiler." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "coordinate-burning", - "metadata": {}, - "outputs": [], - "source": [ - "!pygmentize ./scripts/launch_sm_training_compiler.py" + "**Note:** We recommend you to turn the SageMaker Debugger's profiling and debugging tools off when you use compilation to avoid additional overheads." ] }, { "cell_type": "code", "execution_count": null, - "id": "bacterial-multiple", + "id": "c9352b3b", "metadata": {}, "outputs": [], "source": [ "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", "\n", - "# To use SageMaker Training Compiler in a Distributed setting, please use a wrapper script to invoke your training script\n", - "hyperparameters[\"training_script\"] = f\"run_{LANGUAGE_MODELING_LOSS}.py\"\n", + "# To Enable SageMaker Training Compiler in a Distributed setting\n", + "hyperparameters[\"sagemaker_pytorch_xla_multi_worker_enabled\"] = True\n", "\n", "# with SageMaker Training Compiler we are able to fit a larger batch into memory\n", - "hyperparameters[\"per_device_train_batch_size\"] = 20\n", - "\n", - "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", - "hyperparameters[\"learning_rate\"] = (\n", - " float(\"5e-5\")\n", - " / 32\n", - " * hyperparameters[\"per_device_train_batch_size\"]\n", - " * num_gpus_per_instance\n", - " * NUM_INSTANCES\n", - ")\n", + "hyperparameters[\"per_device_train_batch_size\"] = 84\n", "\n", "# configure the training job\n", "optimized_estimator = HuggingFace(\n", - " entry_point=\"launch_sm_training_compiler.py\", # Wrapper around training script that enables multi node training\n", + " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\", # Wrapper around training script that enables multi node training\n", " compiler_config=TrainingCompilerConfig(), # We are enabling SageMaker Training Compiler here !\n", " source_dir=\"./scripts\",\n", " instance_type=INSTANCE_TYPE,\n", " instance_count=NUM_INSTANCES,\n", " role=role,\n", - " volume_size=200,\n", + " volume_size=512,\n", " py_version=\"py38\",\n", - " transformers_version=\"4.11.0\",\n", - " pytorch_version=\"1.9.0\",\n", + " transformers_version=\"4.21.1\",\n", + " pytorch_version=\"1.11.0\",\n", " hyperparameters=hyperparameters,\n", " disable_profiler=True, # Disable SageMaker Profiler to avoid overhead during benchmarking\n", " debugger_hook_config=False, # Disable SageMaker Debugger to avoid overhead during benchmarking\n", @@ -356,7 +326,7 @@ }, { "cell_type": "markdown", - "id": "regular-country", + "id": "11a94526", "metadata": {}, "source": [ "### Wait for training jobs to complete\n" @@ -365,7 +335,7 @@ { "cell_type": "code", "execution_count": null, - "id": "specific-afternoon", + "id": "f0491c62", "metadata": {}, "outputs": [], "source": [ @@ -381,7 +351,7 @@ }, { "cell_type": "markdown", - "id": "hindu-seven", + "id": "982f1ced", "metadata": {}, "source": [ "## Analysis" @@ -389,7 +359,7 @@ }, { "cell_type": "markdown", - "id": "tender-moral", + "id": "e5c844cc", "metadata": {}, "source": [ "**Note:** If the estimator object is no longer available due to a kernel break or refresh, you need to directly use the training job name and manually attach the training job to a new HuggingFace estimator. For example:\n", @@ -401,7 +371,7 @@ }, { "cell_type": "markdown", - "id": "measured-contributor", + "id": "5a78b433", "metadata": {}, "source": [ "### Load logs of the training job *with* SageMaker Training Compiler" @@ -410,7 +380,7 @@ { "cell_type": "code", "execution_count": null, - "id": "southwest-radio", + "id": "58a4e12b", "metadata": {}, "outputs": [], "source": [ @@ -422,7 +392,7 @@ }, { "cell_type": "markdown", - "id": "numerous-sunglasses", + "id": "fe290d9e", "metadata": {}, "source": [ "### Load logs of the training job *without* SageMaker Training Compiler" @@ -431,7 +401,7 @@ { "cell_type": "code", "execution_count": null, - "id": "eleven-package", + "id": "9b235bff", "metadata": {}, "outputs": [], "source": [ @@ -443,7 +413,7 @@ }, { "cell_type": "markdown", - "id": "metric-cheat", + "id": "b7745278", "metadata": {}, "source": [ "### Create helper functions for analysis" @@ -452,7 +422,7 @@ { "cell_type": "code", "execution_count": null, - "id": "random-party", + "id": "0a10cdbb", "metadata": {}, "outputs": [], "source": [ @@ -493,7 +463,7 @@ }, { "cell_type": "markdown", - "id": "immune-election", + "id": "fc99323f", "metadata": {}, "source": [ "### Plot Optimized vs Native Training Throughput\n", @@ -504,7 +474,7 @@ { "cell_type": "code", "execution_count": null, - "id": "speaking-quantity", + "id": "62df4edc", "metadata": { "scrolled": true }, @@ -525,7 +495,7 @@ { "cell_type": "code", "execution_count": null, - "id": "simple-senate", + "id": "2f118eaa", "metadata": {}, "outputs": [], "source": [ @@ -543,7 +513,7 @@ }, { "cell_type": "markdown", - "id": "ceramic-exception", + "id": "4445e021", "metadata": {}, "source": [ "### Convergence of Training Loss\n", @@ -554,7 +524,7 @@ { "cell_type": "code", "execution_count": null, - "id": "boolean-induction", + "id": "c5fba78c", "metadata": {}, "outputs": [], "source": [ @@ -573,7 +543,7 @@ }, { "cell_type": "markdown", - "id": "promotional-glass", + "id": "01ad7a8e", "metadata": {}, "source": [ "### Training Stats\n", @@ -585,17 +555,19 @@ { "cell_type": "code", "execution_count": null, - "id": "thick-salvation", + "id": "c8e06d6b", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "\n", "pd.DataFrame([n[\"summary\"], o[\"summary\"]], index=[\"Native\", \"Compiler-enhanced\"])" ] }, { "cell_type": "code", "execution_count": null, - "id": "changing-leave", + "id": "85bdc260", "metadata": {}, "outputs": [], "source": [ @@ -611,7 +583,7 @@ }, { "cell_type": "markdown", - "id": "pacific-syria", + "id": "d1f84f0d", "metadata": {}, "source": [ "### Total Billable Time\n", @@ -622,7 +594,7 @@ { "cell_type": "code", "execution_count": null, - "id": "developed-bride", + "id": "b72f6d56", "metadata": {}, "outputs": [], "source": [ @@ -637,7 +609,7 @@ { "cell_type": "code", "execution_count": null, - "id": "interior-landscape", + "id": "db6e4fa6", "metadata": {}, "outputs": [], "source": [ @@ -650,7 +622,7 @@ { "cell_type": "code", "execution_count": null, - "id": "paperback-trash", + "id": "76a933bf", "metadata": {}, "outputs": [], "source": [ @@ -660,7 +632,7 @@ }, { "cell_type": "markdown", - "id": "assured-spread", + "id": "651bd605", "metadata": {}, "source": [ "## Clean up\n", @@ -671,7 +643,7 @@ { "cell_type": "code", "execution_count": null, - "id": "mathematical-islam", + "id": "0b96028a", "metadata": {}, "outputs": [], "source": [ @@ -692,7 +664,7 @@ }, { "cell_type": "markdown", - "id": "grand-refrigerator", + "id": "20e9bd04", "metadata": {}, "source": [ "Also, to find instructions on cleaning up resources, see [Clean Up](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html) in the *Amazon SageMaker Developer Guide*." @@ -701,9 +673,9 @@ ], "metadata": { "kernelspec": { - "display_name": "conda_pytorch_p36", + "display_name": "conda_pytorch_p38", "language": "python", - "name": "conda_pytorch_p36" + "name": "conda_pytorch_p38" }, "language_info": { "codemirror_mode": { @@ -715,7 +687,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.13" + "version": "3.8.12" } }, "nbformat": 4, diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/launch_sm_training_compiler.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/launch_sm_training_compiler.py deleted file mode 100644 index 655af389a2..0000000000 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/launch_sm_training_compiler.py +++ /dev/null @@ -1,9 +0,0 @@ -import subprocess -import sys - -if __name__ == "__main__": - arguments_command = " ".join([arg for arg in sys.argv[1:]]) - """ - The following line will take care of setting up inter node communication as well as managing intra node workers for each GPU. - """ - subprocess.check_call("python -m torch_xla.distributed.sm_dist " + arguments_command, shell=True) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/requirements.txt b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/requirements.txt new file mode 100644 index 0000000000..db6140254d --- /dev/null +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/requirements.txt @@ -0,0 +1,2 @@ +transformers == 4.21.1 +datasets == 1.18.4 \ No newline at end of file diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py new file mode 100644 index 0000000000..a3e13e6f05 --- /dev/null +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py @@ -0,0 +1,582 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2020 The HuggingFace Inc. team. All rights reserved. +# Modifications Copyright 2021 Amazon.com, Inc. or its affiliates. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset. + +Here is the full list of checkpoints on the hub that can be fine-tuned by this script: +https://huggingface.co/models?filter=causal-lm +""" +# You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments. + +import logging +import math +import os +import sys +from dataclasses import dataclass, field +from typing import Optional + +import datasets +from datasets import load_dataset + +import transformers +from transformers import ( + CONFIG_MAPPING, + MODEL_FOR_CAUSAL_LM_MAPPING, + AutoConfig, + AutoModelForCausalLM, + AutoTokenizer, + HfArgumentParser, + Trainer, + TrainingArguments, + default_data_collator, + set_seed, +) +from transformers.trainer_utils import get_last_checkpoint +from transformers.utils import check_min_version +from transformers.utils.versions import require_version + +# for enable herring +import smdistributed.dataparallel.torch.torch_smddp + +# Will error if the minimal version of Transformers is not installed. Remove at your own risks. +check_min_version("4.10.0") + +require_version( + "datasets>=1.8.0", + "To fix: pip install -r examples/pytorch/language-modeling/requirements.txt", +) + +logger = logging.getLogger(__name__) + + +MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys()) +MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES) + + +@dataclass +class ModelArguments: + """ + Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. + """ + + model_name_or_path: Optional[str] = field( + default=None, + metadata={ + "help": "The model checkpoint for weights initialization." + "Don't set if you want to train a model from scratch." + }, + ) + model_type: Optional[str] = field( + default=None, + metadata={ + "help": "If training from scratch, pass a model type from the list: " + + ", ".join(MODEL_TYPES) + }, + ) + config_overrides: Optional[str] = field( + default=None, + metadata={ + "help": "Override some existing default config settings when a model is trained from scratch. Example: " + "n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index" + }, + ) + config_name: Optional[str] = field( + default=None, + metadata={"help": "Pretrained config name or path if not the same as model_name"}, + ) + tokenizer_name: Optional[str] = field( + default=None, + metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}, + ) + cache_dir: Optional[str] = field( + default=None, + metadata={ + "help": "Where do you want to store the pretrained models downloaded from huggingface.co" + }, + ) + use_fast_tokenizer: bool = field( + default=True, + metadata={ + "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not." + }, + ) + model_revision: str = field( + default="main", + metadata={ + "help": "The specific model version to use (can be a branch name, tag name or commit id)." + }, + ) + use_auth_token: bool = field( + default=False, + metadata={ + "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " + "with private models)." + }, + ) + + def __post_init__(self): + if self.config_overrides is not None and ( + self.config_name is not None or self.model_name_or_path is not None + ): + raise ValueError( + "--config_overrides can't be used in combination with --config_name or --model_name_or_path" + ) + + +@dataclass +class DataTrainingArguments: + """ + Arguments pertaining to what data we are going to input our model for training and eval. + """ + + dataset_name: Optional[str] = field( + default=None, + metadata={"help": "The name of the dataset to use (via the datasets library)."}, + ) + dataset_config_name: Optional[str] = field( + default=None, + metadata={ + "help": "The configuration name of the dataset to use (via the datasets library)." + }, + ) + train_file: Optional[str] = field( + default=None, metadata={"help": "The input training data file (a text file)."} + ) + validation_file: Optional[str] = field( + default=None, + metadata={ + "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)." + }, + ) + max_train_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of training examples to this " + "value if set." + }, + ) + max_eval_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this " + "value if set." + }, + ) + + block_size: Optional[int] = field( + default=None, + metadata={ + "help": "Optional input sequence length after tokenization. " + "The training dataset will be truncated in block of this size for training. " + "Default to the model max input length for single sentence inputs (take into account special tokens)." + }, + ) + overwrite_cache: bool = field( + default=False, + metadata={"help": "Overwrite the cached training and evaluation sets"}, + ) + validation_split_percentage: Optional[int] = field( + default=5, + metadata={ + "help": "The percentage of the train set used as validation set in case there's no validation split" + }, + ) + preprocessing_num_workers: Optional[int] = field( + default=None, + metadata={"help": "The number of processes to use for the preprocessing."}, + ) + keep_linebreaks: bool = field( + default=True, + metadata={"help": "Whether to keep line breaks when using TXT files or not."}, + ) + + def __post_init__(self): + if self.dataset_name is None and self.train_file is None and self.validation_file is None: + raise ValueError("Need either a dataset name or a training/validation file.") + else: + if self.train_file is not None: + extension = self.train_file.split(".")[-1] + assert extension in [ + "csv", + "json", + "txt", + ], "`train_file` should be a csv, a json or a txt file." + if self.validation_file is not None: + extension = self.validation_file.split(".")[-1] + assert extension in [ + "csv", + "json", + "txt", + ], "`validation_file` should be a csv, a json or a txt file." + + +def main(): + # See all possible arguments in src/transformers/training_args.py + # or by passing the --help flag to this script. + # We now keep distinct sets of args, for a cleaner separation of concerns. + + parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments)) + if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): + # If we pass only one argument to the script and it's the path to a json file, + # let's parse it to get our arguments. + model_args, data_args, training_args = parser.parse_json_file( + json_file=os.path.abspath(sys.argv[1]) + ) + else: + model_args, data_args, training_args = parser.parse_args_into_dataclasses() + + # Setup logging + logging.basicConfig( + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", + datefmt="%m/%d/%Y %H:%M:%S", + handlers=[logging.StreamHandler(sys.stdout)], + ) + + log_level = training_args.get_process_log_level() + logger.setLevel(log_level) + datasets.utils.logging.set_verbosity(log_level) + transformers.utils.logging.set_verbosity(log_level) + transformers.utils.logging.enable_default_handler() + transformers.utils.logging.enable_explicit_format() + + # Log on each process the small summary: + logger.warning( + f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" + + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" + ) + logger.info(f"Training/evaluation parameters {training_args}") + + # Detecting last checkpoint. + last_checkpoint = None + if ( + os.path.isdir(training_args.output_dir) + and training_args.do_train + and not training_args.overwrite_output_dir + ): + last_checkpoint = get_last_checkpoint(training_args.output_dir) + if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: + raise ValueError( + f"Output directory ({training_args.output_dir}) already exists and is not empty. " + "Use --overwrite_output_dir to overcome." + ) + elif last_checkpoint is not None and training_args.resume_from_checkpoint is None: + logger.info( + f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change " + "the `--output_dir` or add `--overwrite_output_dir` to train from scratch." + ) + + # Set seed before initializing model. + set_seed(training_args.seed) + + # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) + # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ + # (the dataset will be downloaded automatically from the datasets Hub). + # + # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called + # 'text' is found. You can easily tweak this behavior (see below). + # + # In distributed training, the load_dataset function guarantee that only one local process can concurrently + # download the dataset. + if data_args.dataset_name is not None: + # Downloading and loading a dataset from the hub. + raw_datasets = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + cache_dir=model_args.cache_dir, + ) + if "validation" not in raw_datasets.keys(): + raw_datasets["validation"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=f"train[:{data_args.validation_split_percentage}%]", + cache_dir=model_args.cache_dir, + ) + raw_datasets["train"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=f"train[{data_args.validation_split_percentage}%:]", + cache_dir=model_args.cache_dir, + ) + else: + data_files = {} + dataset_args = {} + if data_args.train_file is not None: + data_files["train"] = data_args.train_file + if data_args.validation_file is not None: + data_files["validation"] = data_args.validation_file + extension = ( + data_args.train_file.split(".")[-1] + if data_args.train_file is not None + else data_args.validation_file.split(".")[-1] + ) + if extension == "txt": + extension = "text" + dataset_args["keep_linebreaks"] = data_args.keep_linebreaks + raw_datasets = load_dataset( + extension, + data_files=data_files, + cache_dir=model_args.cache_dir, + **dataset_args, + ) + # If no validation data is there, validation_split_percentage will be used to divide the dataset. + if "validation" not in raw_datasets.keys(): + raw_datasets["validation"] = load_dataset( + extension, + data_files=data_files, + split=f"train[:{data_args.validation_split_percentage}%]", + cache_dir=model_args.cache_dir, + **dataset_args, + ) + raw_datasets["train"] = load_dataset( + extension, + data_files=data_files, + split=f"train[{data_args.validation_split_percentage}%:]", + cache_dir=model_args.cache_dir, + **dataset_args, + ) + + # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at + # https://huggingface.co/docs/datasets/loading_datasets.html. + + # Load pretrained model and tokenizer + # + # Distributed training: + # The .from_pretrained methods guarantee that only one local process can concurrently + # download model & vocab. + + config_kwargs = { + "cache_dir": model_args.cache_dir, + "revision": model_args.model_revision, + "use_auth_token": True if model_args.use_auth_token else None, + } + if model_args.config_name: + config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs) + elif model_args.model_name_or_path: + config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs) + else: + config = CONFIG_MAPPING[model_args.model_type]() + logger.warning("You are instantiating a new config instance from scratch.") + if model_args.config_overrides is not None: + logger.info(f"Overriding config: {model_args.config_overrides}") + config.update_from_string(model_args.config_overrides) + + tokenizer_kwargs = { + "cache_dir": model_args.cache_dir, + "use_fast": model_args.use_fast_tokenizer, + "revision": model_args.model_revision, + "use_auth_token": True if model_args.use_auth_token else None, + } + if model_args.tokenizer_name: + tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs) + elif model_args.model_name_or_path: + tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs) + else: + raise ValueError( + "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You can do it from another script, save it, and load it from here, using --tokenizer_name." + ) + + if model_args.model_name_or_path: + model = AutoModelForCausalLM.from_pretrained( + model_args.model_name_or_path, + from_tf=bool(".ckpt" in model_args.model_name_or_path), + config=config, + cache_dir=model_args.cache_dir, + revision=model_args.model_revision, + use_auth_token=True if model_args.use_auth_token else None, + ) + else: + model = AutoModelForCausalLM.from_config(config) + n_params = sum(dict((p.data_ptr(), p.numel()) for p in model.parameters()).values()) + logger.info(f"Training new model from scratch - Total size={n_params/2**20:.2f}M params") + + model.resize_token_embeddings(len(tokenizer)) + + # Preprocessing the datasets. + # First we tokenize all the texts. + if training_args.do_train: + column_names = raw_datasets["train"].column_names + else: + column_names = raw_datasets["validation"].column_names + text_column_name = "text" if "text" in column_names else column_names[0] + + # since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function + tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base") + + def tokenize_function(examples): + from transformers.testing_utils import CaptureLogger + + with CaptureLogger(tok_logger) as cl: + output = tokenizer(examples[text_column_name]) + # clm input could be much much longer than block_size + if "Token indices sequence length is longer than the" in cl.out: + tok_logger.warning( + "^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model." + ) + return output + + with training_args.main_process_first(desc="dataset map tokenization"): + tokenized_datasets = raw_datasets.map( + tokenize_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on dataset", + ) + + if data_args.block_size is None: + block_size = tokenizer.model_max_length + if block_size > 1024: + logger.warning( + f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " + "Picking 1024 instead. You can change that default value by passing --block_size xxx." + ) + block_size = 1024 + else: + if data_args.block_size > tokenizer.model_max_length: + logger.warning( + f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" + f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." + ) + block_size = min(data_args.block_size, tokenizer.model_max_length) + + # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size. + def group_texts(examples): + # Concatenate all texts. + concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()} + total_length = len(concatenated_examples[list(examples.keys())[0]]) + # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can + # customize this part to your needs. + if total_length >= block_size: + total_length = (total_length // block_size) * block_size + # Split by chunks of max_len. + result = { + k: [t[i : i + block_size] for i in range(0, total_length, block_size)] + for k, t in concatenated_examples.items() + } + result["labels"] = result["input_ids"].copy() + return result + + # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder + # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower + # to preprocess. + # + # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: + # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + + with training_args.main_process_first(desc="grouping texts together"): + lm_datasets = tokenized_datasets.map( + group_texts, + batched=True, + num_proc=data_args.preprocessing_num_workers, + load_from_cache_file=not data_args.overwrite_cache, + desc=f"Grouping texts in chunks of {block_size}", + ) + + if training_args.do_train: + if "train" not in tokenized_datasets: + raise ValueError("--do_train requires a train dataset") + train_dataset = lm_datasets["train"] + if data_args.max_train_samples is not None: + train_dataset = train_dataset.select(range(data_args.max_train_samples)) + + if training_args.do_eval: + if "validation" not in tokenized_datasets: + raise ValueError("--do_eval requires a validation dataset") + eval_dataset = lm_datasets["validation"] + if data_args.max_eval_samples is not None: + eval_dataset = eval_dataset.select(range(data_args.max_eval_samples)) + + # Initialize our Trainer + trainer = Trainer( + model=model, + args=training_args, + train_dataset=train_dataset if training_args.do_train else None, + eval_dataset=eval_dataset if training_args.do_eval else None, + tokenizer=tokenizer, + # Data collator will default to DataCollatorWithPadding, so we change it. + data_collator=default_data_collator, + ) + + # Training + if training_args.do_train: + checkpoint = None + if training_args.resume_from_checkpoint is not None: + checkpoint = training_args.resume_from_checkpoint + elif last_checkpoint is not None: + checkpoint = last_checkpoint + train_result = trainer.train(resume_from_checkpoint=checkpoint) + trainer.save_model() # Saves the tokenizer too for easy upload + + metrics = train_result.metrics + + max_train_samples = ( + data_args.max_train_samples + if data_args.max_train_samples is not None + else len(train_dataset) + ) + metrics["train_samples"] = min(max_train_samples, len(train_dataset)) + + trainer.log_metrics("train", metrics) + trainer.save_metrics("train", metrics) + trainer.save_state() + + # Evaluation + if training_args.do_eval: + logger.info("*** Evaluate ***") + + metrics = trainer.evaluate() + + max_eval_samples = ( + data_args.max_eval_samples + if data_args.max_eval_samples is not None + else len(eval_dataset) + ) + metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset)) + try: + perplexity = math.exp(metrics["eval_loss"]) + except OverflowError: + perplexity = float("inf") + metrics["perplexity"] = perplexity + + trainer.log_metrics("eval", metrics) + trainer.save_metrics("eval", metrics) + + if training_args.push_to_hub: + kwargs = { + "finetuned_from": model_args.model_name_or_path, + "tasks": "text-generation", + } + if data_args.dataset_name is not None: + kwargs["dataset_tags"] = data_args.dataset_name + if data_args.dataset_config_name is not None: + kwargs["dataset_args"] = data_args.dataset_config_name + kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}" + else: + kwargs["dataset"] = data_args.dataset_name + + trainer.push_to_hub(**kwargs) + + +def _mp_fn(index): + # For xla_spawn (TPUs) + main() + + +if __name__ == "__main__": + main() diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py new file mode 100644 index 0000000000..61020a81cb --- /dev/null +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py @@ -0,0 +1,607 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2020 The HuggingFace Team All rights reserved. +# Modifications Copyright 2021 Amazon.com, Inc. or its affiliates. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Fine-tuning the library models for masked language modeling (BERT, ALBERT, RoBERTa...) on a text file or a dataset. + +Here is the full list of checkpoints on the hub that can be fine-tuned by this script: +https://huggingface.co/models?filter=masked-lm +""" +# You can also adapt this script on your own masked language modeling task. Pointers for this are left as comments. + +import logging +import math +import os +import sys +from dataclasses import dataclass, field +from typing import Optional + +import datasets +from datasets import load_dataset + +import transformers +from transformers import ( + CONFIG_MAPPING, + MODEL_FOR_MASKED_LM_MAPPING, + AutoConfig, + AutoModelForMaskedLM, + AutoTokenizer, + DataCollatorForLanguageModeling, + HfArgumentParser, + Trainer, + TrainingArguments, + set_seed, +) +from transformers.trainer_utils import get_last_checkpoint +from transformers.utils import check_min_version +from transformers.utils.versions import require_version + +# for enable herring +import smdistributed.dataparallel.torch.torch_smddp + +# Will error if the minimal version of Transformers is not installed. Remove at your own risks. +check_min_version("4.10.0") + +require_version( + "datasets>=1.8.0", + "To fix: pip install -r examples/pytorch/language-modeling/requirements.txt", +) + +logger = logging.getLogger(__name__) +MODEL_CONFIG_CLASSES = list(MODEL_FOR_MASKED_LM_MAPPING.keys()) +MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES) + + +@dataclass +class ModelArguments: + """ + Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. + """ + + model_name_or_path: Optional[str] = field( + default=None, + metadata={ + "help": "The model checkpoint for weights initialization." + "Don't set if you want to train a model from scratch." + }, + ) + model_type: Optional[str] = field( + default=None, + metadata={ + "help": "If training from scratch, pass a model type from the list: " + + ", ".join(MODEL_TYPES) + }, + ) + config_overrides: Optional[str] = field( + default=None, + metadata={ + "help": "Override some existing default config settings when a model is trained from scratch. Example: " + "n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index" + }, + ) + config_name: Optional[str] = field( + default=None, + metadata={"help": "Pretrained config name or path if not the same as model_name"}, + ) + tokenizer_name: Optional[str] = field( + default=None, + metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}, + ) + cache_dir: Optional[str] = field( + default=None, + metadata={ + "help": "Where do you want to store the pretrained models downloaded from huggingface.co" + }, + ) + use_fast_tokenizer: bool = field( + default=True, + metadata={ + "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not." + }, + ) + model_revision: str = field( + default="main", + metadata={ + "help": "The specific model version to use (can be a branch name, tag name or commit id)." + }, + ) + use_auth_token: bool = field( + default=False, + metadata={ + "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " + "with private models)." + }, + ) + + def __post_init__(self): + if self.config_overrides is not None and ( + self.config_name is not None or self.model_name_or_path is not None + ): + raise ValueError( + "--config_overrides can't be used in combination with --config_name or --model_name_or_path" + ) + + +@dataclass +class DataTrainingArguments: + """ + Arguments pertaining to what data we are going to input our model for training and eval. + """ + + dataset_name: Optional[str] = field( + default=None, + metadata={"help": "The name of the dataset to use (via the datasets library)."}, + ) + dataset_config_name: Optional[str] = field( + default=None, + metadata={ + "help": "The configuration name of the dataset to use (via the datasets library)." + }, + ) + train_file: Optional[str] = field( + default=None, metadata={"help": "The input training data file (a text file)."} + ) + validation_file: Optional[str] = field( + default=None, + metadata={ + "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)." + }, + ) + overwrite_cache: bool = field( + default=False, + metadata={"help": "Overwrite the cached training and evaluation sets"}, + ) + validation_split_percentage: Optional[int] = field( + default=5, + metadata={ + "help": "The percentage of the train set used as validation set in case there's no validation split" + }, + ) + max_seq_length: Optional[int] = field( + default=None, + metadata={ + "help": "The maximum total input sequence length after tokenization. Sequences longer " + "than this will be truncated." + }, + ) + preprocessing_num_workers: Optional[int] = field( + default=None, + metadata={"help": "The number of processes to use for the preprocessing."}, + ) + mlm_probability: float = field( + default=0.15, + metadata={"help": "Ratio of tokens to mask for masked language modeling loss"}, + ) + line_by_line: bool = field( + default=False, + metadata={ + "help": "Whether distinct lines of text in the dataset are to be handled as distinct sequences." + }, + ) + pad_to_max_length: bool = field( + default=False, + metadata={ + "help": "Whether to pad all samples to `max_seq_length`. " + "If False, will pad the samples dynamically when batching to the maximum length in the batch." + }, + ) + max_train_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of training examples to this " + "value if set." + }, + ) + max_eval_samples: Optional[int] = field( + default=None, + metadata={ + "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this " + "value if set." + }, + ) + + def __post_init__(self): + if self.dataset_name is None and self.train_file is None and self.validation_file is None: + raise ValueError("Need either a dataset name or a training/validation file.") + else: + if self.train_file is not None: + extension = self.train_file.split(".")[-1] + assert extension in [ + "csv", + "json", + "txt", + ], "`train_file` should be a csv, a json or a txt file." + if self.validation_file is not None: + extension = self.validation_file.split(".")[-1] + assert extension in [ + "csv", + "json", + "txt", + ], "`validation_file` should be a csv, a json or a txt file." + + +def main(): + # See all possible arguments in src/transformers/training_args.py + # or by passing the --help flag to this script. + # We now keep distinct sets of args, for a cleaner separation of concerns. + + parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments)) + if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): + # If we pass only one argument to the script and it's the path to a json file, + # let's parse it to get our arguments. + model_args, data_args, training_args = parser.parse_json_file( + json_file=os.path.abspath(sys.argv[1]) + ) + else: + model_args, data_args, training_args = parser.parse_args_into_dataclasses() + + # Setup logging + logging.basicConfig( + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", + datefmt="%m/%d/%Y %H:%M:%S", + handlers=[logging.StreamHandler(sys.stdout)], + ) + + log_level = training_args.get_process_log_level() + logger.setLevel(log_level) + datasets.utils.logging.set_verbosity(log_level) + transformers.utils.logging.set_verbosity(log_level) + transformers.utils.logging.enable_default_handler() + transformers.utils.logging.enable_explicit_format() + + # Log on each process the small summary: + logger.warning( + f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" + + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" + ) + # Set the verbosity to info of the Transformers logger (on main process only): + logger.info(f"Training/evaluation parameters {training_args}") + + # Detecting last checkpoint. + last_checkpoint = None + if ( + os.path.isdir(training_args.output_dir) + and training_args.do_train + and not training_args.overwrite_output_dir + ): + last_checkpoint = get_last_checkpoint(training_args.output_dir) + if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: + raise ValueError( + f"Output directory ({training_args.output_dir}) already exists and is not empty. " + "Use --overwrite_output_dir to overcome." + ) + elif last_checkpoint is not None and training_args.resume_from_checkpoint is None: + logger.info( + f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change " + "the `--output_dir` or add `--overwrite_output_dir` to train from scratch." + ) + + # Set seed before initializing model. + set_seed(training_args.seed) + + # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) + # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ + # (the dataset will be downloaded automatically from the datasets Hub + # + # For CSV/JSON files, this script will use the column called 'text' or the first column. You can easily tweak this + # behavior (see below) + # + # In distributed training, the load_dataset function guarantee that only one local process can concurrently + # download the dataset. + if data_args.dataset_name is not None: + # Downloading and loading a dataset from the hub. + raw_datasets = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + cache_dir=model_args.cache_dir, + ) + if "validation" not in raw_datasets.keys(): + raw_datasets["validation"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=f"train[:{data_args.validation_split_percentage}%]", + cache_dir=model_args.cache_dir, + ) + raw_datasets["train"] = load_dataset( + data_args.dataset_name, + data_args.dataset_config_name, + split=f"train[{data_args.validation_split_percentage}%:]", + cache_dir=model_args.cache_dir, + ) + else: + data_files = {} + if data_args.train_file is not None: + data_files["train"] = data_args.train_file + extension = data_args.train_file.split(".")[-1] + if data_args.validation_file is not None: + data_files["validation"] = data_args.validation_file + extension = data_args.validation_file.split(".")[-1] + if extension == "txt": + extension = "text" + raw_datasets = load_dataset( + extension, data_files=data_files, cache_dir=model_args.cache_dir + ) + + # If no validation data is there, validation_split_percentage will be used to divide the dataset. + if "validation" not in raw_datasets.keys(): + raw_datasets["validation"] = load_dataset( + extension, + data_files=data_files, + split=f"train[:{data_args.validation_split_percentage}%]", + cache_dir=model_args.cache_dir, + ) + raw_datasets["train"] = load_dataset( + extension, + data_files=data_files, + split=f"train[{data_args.validation_split_percentage}%:]", + cache_dir=model_args.cache_dir, + ) + + # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at + # https://huggingface.co/docs/datasets/loading_datasets.html. + + # Load pretrained model and tokenizer + # + # Distributed training: + # The .from_pretrained methods guarantee that only one local process can concurrently + # download model & vocab. + config_kwargs = { + "cache_dir": model_args.cache_dir, + "revision": model_args.model_revision, + "use_auth_token": True if model_args.use_auth_token else None, + } + if model_args.config_name: + config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs) + elif model_args.model_name_or_path: + config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs) + else: + config = CONFIG_MAPPING[model_args.model_type]() + logger.warning("You are instantiating a new config instance from scratch.") + if model_args.config_overrides is not None: + logger.info(f"Overriding config: {model_args.config_overrides}") + config.update_from_string(model_args.config_overrides) + + tokenizer_kwargs = { + "cache_dir": model_args.cache_dir, + "use_fast": model_args.use_fast_tokenizer, + "revision": model_args.model_revision, + "use_auth_token": True if model_args.use_auth_token else None, + } + if model_args.tokenizer_name: + tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs) + elif model_args.model_name_or_path: + tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs) + else: + raise ValueError( + "You are instantiating a new tokenizer from scratch. This is not supported by this script." + "You can do it from another script, save it, and load it from here, using --tokenizer_name." + ) + + if model_args.model_name_or_path: + model = AutoModelForMaskedLM.from_pretrained( + model_args.model_name_or_path, + from_tf=bool(".ckpt" in model_args.model_name_or_path), + config=config, + cache_dir=model_args.cache_dir, + revision=model_args.model_revision, + use_auth_token=True if model_args.use_auth_token else None, + ) + else: + logger.info("Training new model from scratch") + model = AutoModelForMaskedLM.from_config(config) + + model.resize_token_embeddings(len(tokenizer)) + + # Preprocessing the datasets. + # First we tokenize all the texts. + if training_args.do_train: + column_names = raw_datasets["train"].column_names + else: + column_names = raw_datasets["validation"].column_names + text_column_name = "text" if "text" in column_names else column_names[0] + + if data_args.max_seq_length is None: + max_seq_length = tokenizer.model_max_length + if max_seq_length > 1024: + logger.warning( + f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " + "Picking 1024 instead. You can change that default value by passing --max_seq_length xxx." + ) + max_seq_length = 1024 + else: + if data_args.max_seq_length > tokenizer.model_max_length: + logger.warning( + f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" + f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." + ) + max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) + + if data_args.line_by_line: + # When using line_by_line, we just tokenize each nonempty line. + padding = "max_length" if data_args.pad_to_max_length else False + + def tokenize_function(examples): + # Remove empty lines + examples[text_column_name] = [ + line for line in examples[text_column_name] if len(line) > 0 and not line.isspace() + ] + return tokenizer( + examples[text_column_name], + padding=padding, + truncation=True, + max_length=max_seq_length, + # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it + # receives the `special_tokens_mask`. + return_special_tokens_mask=True, + ) + + with training_args.main_process_first(desc="dataset map tokenization"): + tokenized_datasets = raw_datasets.map( + tokenize_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=[text_column_name], + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on dataset line_by_line", + ) + else: + # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts. + # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more + # efficient when it receives the `special_tokens_mask`. + def tokenize_function(examples): + return tokenizer(examples[text_column_name], return_special_tokens_mask=True) + + with training_args.main_process_first(desc="dataset map tokenization"): + tokenized_datasets = raw_datasets.map( + tokenize_function, + batched=True, + num_proc=data_args.preprocessing_num_workers, + remove_columns=column_names, + load_from_cache_file=not data_args.overwrite_cache, + desc="Running tokenizer on every text in dataset", + ) + + # Main data processing function that will concatenate all texts from our dataset and generate chunks of + # max_seq_length. + def group_texts(examples): + # Concatenate all texts. + concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()} + total_length = len(concatenated_examples[list(examples.keys())[0]]) + # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can + # customize this part to your needs. + if total_length >= max_seq_length: + total_length = (total_length // max_seq_length) * max_seq_length + # Split by chunks of max_len. + result = { + k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)] + for k, t in concatenated_examples.items() + } + return result + + # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a + # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value + # might be slower to preprocess. + # + # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: + # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map + + with training_args.main_process_first(desc="grouping texts together"): + tokenized_datasets = tokenized_datasets.map( + group_texts, + batched=True, + num_proc=data_args.preprocessing_num_workers, + load_from_cache_file=not data_args.overwrite_cache, + desc=f"Grouping texts in chunks of {max_seq_length}", + ) + + if training_args.do_train: + if "train" not in tokenized_datasets: + raise ValueError("--do_train requires a train dataset") + train_dataset = tokenized_datasets["train"] + if data_args.max_train_samples is not None: + train_dataset = train_dataset.select(range(data_args.max_train_samples)) + + if training_args.do_eval: + if "validation" not in tokenized_datasets: + raise ValueError("--do_eval requires a validation dataset") + eval_dataset = tokenized_datasets["validation"] + if data_args.max_eval_samples is not None: + eval_dataset = eval_dataset.select(range(data_args.max_eval_samples)) + + # Data collator + # This one will take care of randomly masking the tokens. + pad_to_multiple_of_8 = ( + data_args.line_by_line and training_args.fp16 and not data_args.pad_to_max_length + ) + data_collator = DataCollatorForLanguageModeling( + tokenizer=tokenizer, + mlm_probability=data_args.mlm_probability, + pad_to_multiple_of=8 if pad_to_multiple_of_8 else None, + ) + + # Initialize our Trainer + trainer = Trainer( + model=model, + args=training_args, + train_dataset=train_dataset if training_args.do_train else None, + eval_dataset=eval_dataset if training_args.do_eval else None, + tokenizer=tokenizer, + data_collator=data_collator, + ) + + # Training + if training_args.do_train: + checkpoint = None + if training_args.resume_from_checkpoint is not None: + checkpoint = training_args.resume_from_checkpoint + elif last_checkpoint is not None: + checkpoint = last_checkpoint + train_result = trainer.train(resume_from_checkpoint=checkpoint) + trainer.save_model() # Saves the tokenizer too for easy upload + metrics = train_result.metrics + + max_train_samples = ( + data_args.max_train_samples + if data_args.max_train_samples is not None + else len(train_dataset) + ) + metrics["train_samples"] = min(max_train_samples, len(train_dataset)) + + trainer.log_metrics("train", metrics) + trainer.save_metrics("train", metrics) + trainer.save_state() + + # Evaluation + if training_args.do_eval: + logger.info("*** Evaluate ***") + + metrics = trainer.evaluate() + + max_eval_samples = ( + data_args.max_eval_samples + if data_args.max_eval_samples is not None + else len(eval_dataset) + ) + metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset)) + try: + perplexity = math.exp(metrics["eval_loss"]) + except OverflowError: + perplexity = float("inf") + metrics["perplexity"] = perplexity + + trainer.log_metrics("eval", metrics) + trainer.save_metrics("eval", metrics) + + if training_args.push_to_hub: + kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "fill-mask"} + if data_args.dataset_name is not None: + kwargs["dataset_tags"] = data_args.dataset_name + if data_args.dataset_config_name is not None: + kwargs["dataset_args"] = data_args.dataset_config_name + kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}" + else: + kwargs["dataset"] = data_args.dataset_name + + trainer.push_to_hub(**kwargs) + + +def _mp_fn(index): + # For xla_spawn (TPUs) + main() + + +if __name__ == "__main__": + main() From b22f3767dbb5756716248d7b78a11b5d8d8a2bcd Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Thu, 6 Oct 2022 21:18:24 +0000 Subject: [PATCH 2/9] edit parameters for estimators --- ...nguage-modeling-multi-gpu-multi-node.ipynb | 200 +++--- .../scripts/run_clm_smddp.py | 582 ----------------- .../scripts/run_mlm_smddp.py | 607 ------------------ 3 files changed, 117 insertions(+), 1272 deletions(-) delete mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py delete mode 100644 sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index d3f93409d4..31458e6a9f 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "3e95cc90", + "id": "3c17b89c", "metadata": {}, "source": [ "# Compile and Train the GPT2 Model using the Transformers Trainer API with the SST2 Dataset for Multi-Node Multi-GPU Training" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "444fe156", + "id": "297eba0a", "metadata": {}, "source": [ "1. [Introduction](#Introduction) \n", @@ -25,7 +25,7 @@ }, { "cell_type": "markdown", - "id": "4bf9ceb2", + "id": "32474760", "metadata": {}, "source": [ "## SageMaker Training Compiler Overview\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "324b417c", + "id": "a76ece43", "metadata": {}, "source": [ "## Development Environment " @@ -55,7 +55,7 @@ }, { "cell_type": "markdown", - "id": "f6ef966d", + "id": "62fe1493", "metadata": {}, "source": [ "### Installation\n", @@ -66,7 +66,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a646c234", + "id": "1a3240eb", "metadata": {}, "outputs": [], "source": [ @@ -76,7 +76,7 @@ { "cell_type": "code", "execution_count": null, - "id": "91ecb4bb", + "id": "a2b9276e", "metadata": {}, "outputs": [], "source": [ @@ -86,7 +86,7 @@ { "cell_type": "code", "execution_count": null, - "id": "56fb1066", + "id": "824bc2b5", "metadata": {}, "outputs": [], "source": [ @@ -102,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "5a03c1ef", + "id": "e97e9356", "metadata": {}, "source": [ "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for datasets library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "814b7ca3", + "id": "201cc329", "metadata": {}, "source": [ "### SageMaker environment " @@ -127,7 +127,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ad39a3f6", + "id": "76b34d36", "metadata": {}, "outputs": [], "source": [ @@ -152,7 +152,7 @@ }, { "cell_type": "markdown", - "id": "55e4e7eb", + "id": "06e57ac1", "metadata": {}, "source": [ "## SageMaker Training Job\n", @@ -166,7 +166,7 @@ }, { "cell_type": "markdown", - "id": "5cdf928c", + "id": "0b5f2232", "metadata": {}, "source": [ "### Training Setup" @@ -175,13 +175,13 @@ { "cell_type": "code", "execution_count": null, - "id": "6a87f6e7", + "id": "7f1b2444", "metadata": {}, "outputs": [], "source": [ "# Here we configure the training job. Please configure the appropriate options below:\n", "\n", - "EPOCHS = 400\n", + "EPOCHS = 600\n", "\n", "# Choose between Causal Language Model and Masked Language Model\n", "LANGUAGE_MODELING_LOSS = \"clm\" # or \"mlm\"\n", @@ -194,16 +194,43 @@ "\n", "# For more information about the options, please look into the training scripts\n", "\n", - "# SageMaker Training Compiler currently only supports training on GPU\n", "# Select Instance type for training\n", - "INSTANCE_TYPE = \"ml.p4d.24xlarge\" # ml.p3.8xlarge is easily available. However, ml.p3.16xlarge provides better performance.\n", + "INSTANCE_TYPE = \"ml.p4d.24xlarge\" \n", "NUM_INSTANCES = 2\n", - "num_gpus_per_instance = 8" + "# Since ml.p4d.24xlarge instance has 8 GPUs, we set num_gpus_per_instance to 8\n", + "num_gpus_per_instance = 8\n", + "\n", + "\n", + "\n", + "estimator_args = dict(\n", + " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\",\n", + " source_dir=\"./scripts\",\n", + " instance_type=INSTANCE_TYPE,\n", + " instance_count=NUM_INSTANCES,\n", + " role=role,\n", + " volume_size=100,\n", + " py_version=\"py38\",\n", + " disable_profiler=True, # Disabling SageMaker Profiler to avoid overheads during benchmarking\n", + " debugger_hook_config=False, # Disabling SageMaker Debugger to avoid overheads during benchmarking\n", + " base_job_name=\"trcomp-pt-example\",\n", + " metric_definitions=[\n", + " {\"Name\": \"summary_train_runtime\", \"Regex\": \"'train_runtime': ([0-9.]*)\"},\n", + " {\n", + " \"Name\": \"summary_train_samples_per_second\",\n", + " \"Regex\": \"'train_samples_per_second': ([0-9.]*)\",\n", + " },\n", + " {\"Name\": \"summary_train_steps_per_second\", \"Regex\": \"'train_steps_per_second': ([0-9.]*)\"},\n", + " {\"Name\": \"summary_train_loss\", \"Regex\": \"'train_loss': ([0-9.]*)\"},\n", + " {\"Name\": \"epoch\", \"Regex\": \"'epoch': ([0-9.]*)\"},\n", + " {\"Name\": \"train_loss\", \"Regex\": \"'loss': ([0-9.]*)\"},\n", + " {\"Name\": \"learning_rate\", \"Regex\": \"'learning_rate': ([0-9.]*)\"},\n", + " ], \n", + ")" ] }, { "cell_type": "markdown", - "id": "630bed45", + "id": "ac0f0182", "metadata": {}, "source": [ "### Training with SageMaker Distributed Data Parallel " @@ -211,16 +238,16 @@ }, { "cell_type": "markdown", - "id": "0af1f969", + "id": "3dbfd6a1", "metadata": {}, "source": [ - "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use AMP for faster training." + "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training." ] }, { "cell_type": "code", "execution_count": null, - "id": "8f8df334", + "id": "187569b9", "metadata": {}, "outputs": [], "source": [ @@ -233,35 +260,44 @@ " \"dataset_name\": \"glue\",\n", " \"dataset_config_name\": \"sst2\",\n", " \"do_train\": True,\n", - " \"do_eval\": True,\n", + " \"do_eval\": False,\n", " \"fp16\": True,\n", " \"per_device_train_batch_size\": 32,\n", - " \"overwrite_output_dir\": True,\n", " \"num_train_epochs\": EPOCHS,\n", + " \"overwrite_output_dir\": True,\n", + " \"evaluation_strategy\": \"no\",\n", + " \"logging_strategy\": \"epoch\",\n", " \"output_dir\": \"/opt/ml/model\",\n", + " \"dataloader_drop_last\": True,\n", " SEQ_LEN_ARG: 512,\n", - " \"logging_strategy\": \"epoch\",\n", - " \"save_strategy\": \"no\",\n", " \"preprocessing_num_workers\": 12,\n", - "}\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4afd4ae", + "metadata": {}, + "outputs": [], + "source": [ + "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", + "hyperparameters[\"learning_rate\"] = (\n", + " float(\"5e-5\")\n", + " / 32\n", + " * hyperparameters[\"per_device_train_batch_size\"]\n", + " * num_gpus_per_instance\n", + " * NUM_INSTANCES\n", + ")\n", "\n", "# configure the training job\n", "native_estimator = PyTorch(\n", - " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}_smddp.py\",\n", - " source_dir=\"./scripts\",\n", - " instance_type=INSTANCE_TYPE,\n", - " instance_count=NUM_INSTANCES,\n", - " role=role,\n", - " volume_size=512,\n", - " transformers_version=\"4.21.1\",\n", - " framework_version=\"1.11.0\",\n", - " py_version=\"py38\",\n", + " **estimator_args,\n", + " framework_version=\"1.11\",\n", " hyperparameters=hyperparameters,\n", " distribution={\n", " \"smdistributed\": {\"dataparallel\": {\"enabled\": True}}\n", " }, # Use SageMaker Data Distributed Parallel to train across nodes/GPUs.\n", - " disable_profiler=True, # Disabling SageMaker Profiler to avoid overhead during benchmarking\n", - " debugger_hook_config=False, # Disabling SageMaker Debugger to avoid overhead during benchmarking\n", ")\n", "\n", "# Start the training job\n", @@ -271,7 +307,7 @@ }, { "cell_type": "markdown", - "id": "98e98391", + "id": "6a9388b4", "metadata": {}, "source": [ "### Training with SageMaker Training Compiler " @@ -279,7 +315,7 @@ }, { "cell_type": "markdown", - "id": "9dbbfdd5", + "id": "5226190d", "metadata": {}, "source": [ "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately.\n", @@ -290,33 +326,32 @@ { "cell_type": "code", "execution_count": null, - "id": "c9352b3b", + "id": "403e0e24", "metadata": {}, "outputs": [], "source": [ "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", "\n", - "# To Enable SageMaker Training Compiler in a Distributed setting\n", - "hyperparameters[\"sagemaker_pytorch_xla_multi_worker_enabled\"] = True\n", - "\n", "# with SageMaker Training Compiler we are able to fit a larger batch into memory\n", - "hyperparameters[\"per_device_train_batch_size\"] = 84\n", + "hyperparameters[\"per_device_train_batch_size\"] = 64\n", + "\n", + "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", + "hyperparameters[\"learning_rate\"] = (\n", + " float(\"5e-5\")\n", + " / 32\n", + " * hyperparameters[\"per_device_train_batch_size\"]\n", + " * num_gpus_per_instance\n", + " * NUM_INSTANCES\n", + ")\n", "\n", "# configure the training job\n", "optimized_estimator = HuggingFace(\n", - " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\", # Wrapper around training script that enables multi node training\n", - " compiler_config=TrainingCompilerConfig(), # We are enabling SageMaker Training Compiler here !\n", - " source_dir=\"./scripts\",\n", - " instance_type=INSTANCE_TYPE,\n", - " instance_count=NUM_INSTANCES,\n", - " role=role,\n", - " volume_size=512,\n", - " py_version=\"py38\",\n", - " transformers_version=\"4.21.1\",\n", - " pytorch_version=\"1.11.0\",\n", + " compiler_config=TrainingCompilerConfig(),\n", + " **estimator_args,\n", + " transformers_version=\"4.21\",\n", + " pytorch_version=\"1.11\",\n", " hyperparameters=hyperparameters,\n", - " disable_profiler=True, # Disable SageMaker Profiler to avoid overhead during benchmarking\n", - " debugger_hook_config=False, # Disable SageMaker Debugger to avoid overhead during benchmarking\n", + " distribution={\"pytorchxla\": {\"enabled\": True}},\n", ")\n", "\n", "# start the training job\n", @@ -326,7 +361,7 @@ }, { "cell_type": "markdown", - "id": "11a94526", + "id": "63403eee", "metadata": {}, "source": [ "### Wait for training jobs to complete\n" @@ -335,7 +370,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f0491c62", + "id": "cd5455f8", "metadata": {}, "outputs": [], "source": [ @@ -351,7 +386,7 @@ }, { "cell_type": "markdown", - "id": "982f1ced", + "id": "13ca3e97", "metadata": {}, "source": [ "## Analysis" @@ -359,7 +394,7 @@ }, { "cell_type": "markdown", - "id": "e5c844cc", + "id": "b05fb71b", "metadata": {}, "source": [ "**Note:** If the estimator object is no longer available due to a kernel break or refresh, you need to directly use the training job name and manually attach the training job to a new HuggingFace estimator. For example:\n", @@ -371,7 +406,7 @@ }, { "cell_type": "markdown", - "id": "5a78b433", + "id": "69097726", "metadata": {}, "source": [ "### Load logs of the training job *with* SageMaker Training Compiler" @@ -380,7 +415,7 @@ { "cell_type": "code", "execution_count": null, - "id": "58a4e12b", + "id": "59e6ebf2", "metadata": {}, "outputs": [], "source": [ @@ -392,7 +427,7 @@ }, { "cell_type": "markdown", - "id": "fe290d9e", + "id": "638edd10", "metadata": {}, "source": [ "### Load logs of the training job *without* SageMaker Training Compiler" @@ -401,7 +436,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9b235bff", + "id": "4d87d7e4", "metadata": {}, "outputs": [], "source": [ @@ -413,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "b7745278", + "id": "69f23ac1", "metadata": {}, "source": [ "### Create helper functions for analysis" @@ -422,7 +457,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0a10cdbb", + "id": "e863252a", "metadata": {}, "outputs": [], "source": [ @@ -463,7 +498,7 @@ }, { "cell_type": "markdown", - "id": "fc99323f", + "id": "901c0346", "metadata": {}, "source": [ "### Plot Optimized vs Native Training Throughput\n", @@ -474,7 +509,7 @@ { "cell_type": "code", "execution_count": null, - "id": "62df4edc", + "id": "1c053bbe", "metadata": { "scrolled": true }, @@ -495,7 +530,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2f118eaa", + "id": "21eea555", "metadata": {}, "outputs": [], "source": [ @@ -513,7 +548,7 @@ }, { "cell_type": "markdown", - "id": "4445e021", + "id": "5c21e6c8", "metadata": {}, "source": [ "### Convergence of Training Loss\n", @@ -524,7 +559,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c5fba78c", + "id": "0734befd", "metadata": {}, "outputs": [], "source": [ @@ -543,7 +578,7 @@ }, { "cell_type": "markdown", - "id": "01ad7a8e", + "id": "323fb548", "metadata": {}, "source": [ "### Training Stats\n", @@ -555,19 +590,18 @@ { "cell_type": "code", "execution_count": null, - "id": "c8e06d6b", + "id": "7f757dc0", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", - "\n", "pd.DataFrame([n[\"summary\"], o[\"summary\"]], index=[\"Native\", \"Compiler-enhanced\"])" ] }, { "cell_type": "code", "execution_count": null, - "id": "85bdc260", + "id": "f45b0aab", "metadata": {}, "outputs": [], "source": [ @@ -583,7 +617,7 @@ }, { "cell_type": "markdown", - "id": "d1f84f0d", + "id": "266f2257", "metadata": {}, "source": [ "### Total Billable Time\n", @@ -594,7 +628,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b72f6d56", + "id": "2a7b17be", "metadata": {}, "outputs": [], "source": [ @@ -609,7 +643,7 @@ { "cell_type": "code", "execution_count": null, - "id": "db6e4fa6", + "id": "e3d3967f", "metadata": {}, "outputs": [], "source": [ @@ -622,7 +656,7 @@ { "cell_type": "code", "execution_count": null, - "id": "76a933bf", + "id": "d5deb4d6", "metadata": {}, "outputs": [], "source": [ @@ -632,7 +666,7 @@ }, { "cell_type": "markdown", - "id": "651bd605", + "id": "0527c59e", "metadata": {}, "source": [ "## Clean up\n", @@ -643,7 +677,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0b96028a", + "id": "7aa8a7dc", "metadata": {}, "outputs": [], "source": [ @@ -664,7 +698,7 @@ }, { "cell_type": "markdown", - "id": "20e9bd04", + "id": "6d48d974", "metadata": {}, "source": [ "Also, to find instructions on cleaning up resources, see [Clean Up](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html) in the *Amazon SageMaker Developer Guide*." diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py deleted file mode 100644 index a3e13e6f05..0000000000 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm_smddp.py +++ /dev/null @@ -1,582 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -# Copyright 2020 The HuggingFace Inc. team. All rights reserved. -# Modifications Copyright 2021 Amazon.com, Inc. or its affiliates. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -""" -Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset. - -Here is the full list of checkpoints on the hub that can be fine-tuned by this script: -https://huggingface.co/models?filter=causal-lm -""" -# You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments. - -import logging -import math -import os -import sys -from dataclasses import dataclass, field -from typing import Optional - -import datasets -from datasets import load_dataset - -import transformers -from transformers import ( - CONFIG_MAPPING, - MODEL_FOR_CAUSAL_LM_MAPPING, - AutoConfig, - AutoModelForCausalLM, - AutoTokenizer, - HfArgumentParser, - Trainer, - TrainingArguments, - default_data_collator, - set_seed, -) -from transformers.trainer_utils import get_last_checkpoint -from transformers.utils import check_min_version -from transformers.utils.versions import require_version - -# for enable herring -import smdistributed.dataparallel.torch.torch_smddp - -# Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.10.0") - -require_version( - "datasets>=1.8.0", - "To fix: pip install -r examples/pytorch/language-modeling/requirements.txt", -) - -logger = logging.getLogger(__name__) - - -MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys()) -MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES) - - -@dataclass -class ModelArguments: - """ - Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. - """ - - model_name_or_path: Optional[str] = field( - default=None, - metadata={ - "help": "The model checkpoint for weights initialization." - "Don't set if you want to train a model from scratch." - }, - ) - model_type: Optional[str] = field( - default=None, - metadata={ - "help": "If training from scratch, pass a model type from the list: " - + ", ".join(MODEL_TYPES) - }, - ) - config_overrides: Optional[str] = field( - default=None, - metadata={ - "help": "Override some existing default config settings when a model is trained from scratch. Example: " - "n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index" - }, - ) - config_name: Optional[str] = field( - default=None, - metadata={"help": "Pretrained config name or path if not the same as model_name"}, - ) - tokenizer_name: Optional[str] = field( - default=None, - metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}, - ) - cache_dir: Optional[str] = field( - default=None, - metadata={ - "help": "Where do you want to store the pretrained models downloaded from huggingface.co" - }, - ) - use_fast_tokenizer: bool = field( - default=True, - metadata={ - "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not." - }, - ) - model_revision: str = field( - default="main", - metadata={ - "help": "The specific model version to use (can be a branch name, tag name or commit id)." - }, - ) - use_auth_token: bool = field( - default=False, - metadata={ - "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " - "with private models)." - }, - ) - - def __post_init__(self): - if self.config_overrides is not None and ( - self.config_name is not None or self.model_name_or_path is not None - ): - raise ValueError( - "--config_overrides can't be used in combination with --config_name or --model_name_or_path" - ) - - -@dataclass -class DataTrainingArguments: - """ - Arguments pertaining to what data we are going to input our model for training and eval. - """ - - dataset_name: Optional[str] = field( - default=None, - metadata={"help": "The name of the dataset to use (via the datasets library)."}, - ) - dataset_config_name: Optional[str] = field( - default=None, - metadata={ - "help": "The configuration name of the dataset to use (via the datasets library)." - }, - ) - train_file: Optional[str] = field( - default=None, metadata={"help": "The input training data file (a text file)."} - ) - validation_file: Optional[str] = field( - default=None, - metadata={ - "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)." - }, - ) - max_train_samples: Optional[int] = field( - default=None, - metadata={ - "help": "For debugging purposes or quicker training, truncate the number of training examples to this " - "value if set." - }, - ) - max_eval_samples: Optional[int] = field( - default=None, - metadata={ - "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this " - "value if set." - }, - ) - - block_size: Optional[int] = field( - default=None, - metadata={ - "help": "Optional input sequence length after tokenization. " - "The training dataset will be truncated in block of this size for training. " - "Default to the model max input length for single sentence inputs (take into account special tokens)." - }, - ) - overwrite_cache: bool = field( - default=False, - metadata={"help": "Overwrite the cached training and evaluation sets"}, - ) - validation_split_percentage: Optional[int] = field( - default=5, - metadata={ - "help": "The percentage of the train set used as validation set in case there's no validation split" - }, - ) - preprocessing_num_workers: Optional[int] = field( - default=None, - metadata={"help": "The number of processes to use for the preprocessing."}, - ) - keep_linebreaks: bool = field( - default=True, - metadata={"help": "Whether to keep line breaks when using TXT files or not."}, - ) - - def __post_init__(self): - if self.dataset_name is None and self.train_file is None and self.validation_file is None: - raise ValueError("Need either a dataset name or a training/validation file.") - else: - if self.train_file is not None: - extension = self.train_file.split(".")[-1] - assert extension in [ - "csv", - "json", - "txt", - ], "`train_file` should be a csv, a json or a txt file." - if self.validation_file is not None: - extension = self.validation_file.split(".")[-1] - assert extension in [ - "csv", - "json", - "txt", - ], "`validation_file` should be a csv, a json or a txt file." - - -def main(): - # See all possible arguments in src/transformers/training_args.py - # or by passing the --help flag to this script. - # We now keep distinct sets of args, for a cleaner separation of concerns. - - parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments)) - if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): - # If we pass only one argument to the script and it's the path to a json file, - # let's parse it to get our arguments. - model_args, data_args, training_args = parser.parse_json_file( - json_file=os.path.abspath(sys.argv[1]) - ) - else: - model_args, data_args, training_args = parser.parse_args_into_dataclasses() - - # Setup logging - logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - handlers=[logging.StreamHandler(sys.stdout)], - ) - - log_level = training_args.get_process_log_level() - logger.setLevel(log_level) - datasets.utils.logging.set_verbosity(log_level) - transformers.utils.logging.set_verbosity(log_level) - transformers.utils.logging.enable_default_handler() - transformers.utils.logging.enable_explicit_format() - - # Log on each process the small summary: - logger.warning( - f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" - + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" - ) - logger.info(f"Training/evaluation parameters {training_args}") - - # Detecting last checkpoint. - last_checkpoint = None - if ( - os.path.isdir(training_args.output_dir) - and training_args.do_train - and not training_args.overwrite_output_dir - ): - last_checkpoint = get_last_checkpoint(training_args.output_dir) - if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: - raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty. " - "Use --overwrite_output_dir to overcome." - ) - elif last_checkpoint is not None and training_args.resume_from_checkpoint is None: - logger.info( - f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change " - "the `--output_dir` or add `--overwrite_output_dir` to train from scratch." - ) - - # Set seed before initializing model. - set_seed(training_args.seed) - - # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) - # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ - # (the dataset will be downloaded automatically from the datasets Hub). - # - # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called - # 'text' is found. You can easily tweak this behavior (see below). - # - # In distributed training, the load_dataset function guarantee that only one local process can concurrently - # download the dataset. - if data_args.dataset_name is not None: - # Downloading and loading a dataset from the hub. - raw_datasets = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - cache_dir=model_args.cache_dir, - ) - if "validation" not in raw_datasets.keys(): - raw_datasets["validation"] = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - split=f"train[:{data_args.validation_split_percentage}%]", - cache_dir=model_args.cache_dir, - ) - raw_datasets["train"] = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - split=f"train[{data_args.validation_split_percentage}%:]", - cache_dir=model_args.cache_dir, - ) - else: - data_files = {} - dataset_args = {} - if data_args.train_file is not None: - data_files["train"] = data_args.train_file - if data_args.validation_file is not None: - data_files["validation"] = data_args.validation_file - extension = ( - data_args.train_file.split(".")[-1] - if data_args.train_file is not None - else data_args.validation_file.split(".")[-1] - ) - if extension == "txt": - extension = "text" - dataset_args["keep_linebreaks"] = data_args.keep_linebreaks - raw_datasets = load_dataset( - extension, - data_files=data_files, - cache_dir=model_args.cache_dir, - **dataset_args, - ) - # If no validation data is there, validation_split_percentage will be used to divide the dataset. - if "validation" not in raw_datasets.keys(): - raw_datasets["validation"] = load_dataset( - extension, - data_files=data_files, - split=f"train[:{data_args.validation_split_percentage}%]", - cache_dir=model_args.cache_dir, - **dataset_args, - ) - raw_datasets["train"] = load_dataset( - extension, - data_files=data_files, - split=f"train[{data_args.validation_split_percentage}%:]", - cache_dir=model_args.cache_dir, - **dataset_args, - ) - - # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. - - # Load pretrained model and tokenizer - # - # Distributed training: - # The .from_pretrained methods guarantee that only one local process can concurrently - # download model & vocab. - - config_kwargs = { - "cache_dir": model_args.cache_dir, - "revision": model_args.model_revision, - "use_auth_token": True if model_args.use_auth_token else None, - } - if model_args.config_name: - config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs) - elif model_args.model_name_or_path: - config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs) - else: - config = CONFIG_MAPPING[model_args.model_type]() - logger.warning("You are instantiating a new config instance from scratch.") - if model_args.config_overrides is not None: - logger.info(f"Overriding config: {model_args.config_overrides}") - config.update_from_string(model_args.config_overrides) - - tokenizer_kwargs = { - "cache_dir": model_args.cache_dir, - "use_fast": model_args.use_fast_tokenizer, - "revision": model_args.model_revision, - "use_auth_token": True if model_args.use_auth_token else None, - } - if model_args.tokenizer_name: - tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs) - elif model_args.model_name_or_path: - tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs) - else: - raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." - "You can do it from another script, save it, and load it from here, using --tokenizer_name." - ) - - if model_args.model_name_or_path: - model = AutoModelForCausalLM.from_pretrained( - model_args.model_name_or_path, - from_tf=bool(".ckpt" in model_args.model_name_or_path), - config=config, - cache_dir=model_args.cache_dir, - revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, - ) - else: - model = AutoModelForCausalLM.from_config(config) - n_params = sum(dict((p.data_ptr(), p.numel()) for p in model.parameters()).values()) - logger.info(f"Training new model from scratch - Total size={n_params/2**20:.2f}M params") - - model.resize_token_embeddings(len(tokenizer)) - - # Preprocessing the datasets. - # First we tokenize all the texts. - if training_args.do_train: - column_names = raw_datasets["train"].column_names - else: - column_names = raw_datasets["validation"].column_names - text_column_name = "text" if "text" in column_names else column_names[0] - - # since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function - tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base") - - def tokenize_function(examples): - from transformers.testing_utils import CaptureLogger - - with CaptureLogger(tok_logger) as cl: - output = tokenizer(examples[text_column_name]) - # clm input could be much much longer than block_size - if "Token indices sequence length is longer than the" in cl.out: - tok_logger.warning( - "^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model." - ) - return output - - with training_args.main_process_first(desc="dataset map tokenization"): - tokenized_datasets = raw_datasets.map( - tokenize_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on dataset", - ) - - if data_args.block_size is None: - block_size = tokenizer.model_max_length - if block_size > 1024: - logger.warning( - f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " - "Picking 1024 instead. You can change that default value by passing --block_size xxx." - ) - block_size = 1024 - else: - if data_args.block_size > tokenizer.model_max_length: - logger.warning( - f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" - f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." - ) - block_size = min(data_args.block_size, tokenizer.model_max_length) - - # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size. - def group_texts(examples): - # Concatenate all texts. - concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()} - total_length = len(concatenated_examples[list(examples.keys())[0]]) - # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can - # customize this part to your needs. - if total_length >= block_size: - total_length = (total_length // block_size) * block_size - # Split by chunks of max_len. - result = { - k: [t[i : i + block_size] for i in range(0, total_length, block_size)] - for k, t in concatenated_examples.items() - } - result["labels"] = result["input_ids"].copy() - return result - - # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder - # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower - # to preprocess. - # - # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map - - with training_args.main_process_first(desc="grouping texts together"): - lm_datasets = tokenized_datasets.map( - group_texts, - batched=True, - num_proc=data_args.preprocessing_num_workers, - load_from_cache_file=not data_args.overwrite_cache, - desc=f"Grouping texts in chunks of {block_size}", - ) - - if training_args.do_train: - if "train" not in tokenized_datasets: - raise ValueError("--do_train requires a train dataset") - train_dataset = lm_datasets["train"] - if data_args.max_train_samples is not None: - train_dataset = train_dataset.select(range(data_args.max_train_samples)) - - if training_args.do_eval: - if "validation" not in tokenized_datasets: - raise ValueError("--do_eval requires a validation dataset") - eval_dataset = lm_datasets["validation"] - if data_args.max_eval_samples is not None: - eval_dataset = eval_dataset.select(range(data_args.max_eval_samples)) - - # Initialize our Trainer - trainer = Trainer( - model=model, - args=training_args, - train_dataset=train_dataset if training_args.do_train else None, - eval_dataset=eval_dataset if training_args.do_eval else None, - tokenizer=tokenizer, - # Data collator will default to DataCollatorWithPadding, so we change it. - data_collator=default_data_collator, - ) - - # Training - if training_args.do_train: - checkpoint = None - if training_args.resume_from_checkpoint is not None: - checkpoint = training_args.resume_from_checkpoint - elif last_checkpoint is not None: - checkpoint = last_checkpoint - train_result = trainer.train(resume_from_checkpoint=checkpoint) - trainer.save_model() # Saves the tokenizer too for easy upload - - metrics = train_result.metrics - - max_train_samples = ( - data_args.max_train_samples - if data_args.max_train_samples is not None - else len(train_dataset) - ) - metrics["train_samples"] = min(max_train_samples, len(train_dataset)) - - trainer.log_metrics("train", metrics) - trainer.save_metrics("train", metrics) - trainer.save_state() - - # Evaluation - if training_args.do_eval: - logger.info("*** Evaluate ***") - - metrics = trainer.evaluate() - - max_eval_samples = ( - data_args.max_eval_samples - if data_args.max_eval_samples is not None - else len(eval_dataset) - ) - metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset)) - try: - perplexity = math.exp(metrics["eval_loss"]) - except OverflowError: - perplexity = float("inf") - metrics["perplexity"] = perplexity - - trainer.log_metrics("eval", metrics) - trainer.save_metrics("eval", metrics) - - if training_args.push_to_hub: - kwargs = { - "finetuned_from": model_args.model_name_or_path, - "tasks": "text-generation", - } - if data_args.dataset_name is not None: - kwargs["dataset_tags"] = data_args.dataset_name - if data_args.dataset_config_name is not None: - kwargs["dataset_args"] = data_args.dataset_config_name - kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}" - else: - kwargs["dataset"] = data_args.dataset_name - - trainer.push_to_hub(**kwargs) - - -def _mp_fn(index): - # For xla_spawn (TPUs) - main() - - -if __name__ == "__main__": - main() diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py deleted file mode 100644 index 61020a81cb..0000000000 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm_smddp.py +++ /dev/null @@ -1,607 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -# Copyright 2020 The HuggingFace Team All rights reserved. -# Modifications Copyright 2021 Amazon.com, Inc. or its affiliates. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -""" -Fine-tuning the library models for masked language modeling (BERT, ALBERT, RoBERTa...) on a text file or a dataset. - -Here is the full list of checkpoints on the hub that can be fine-tuned by this script: -https://huggingface.co/models?filter=masked-lm -""" -# You can also adapt this script on your own masked language modeling task. Pointers for this are left as comments. - -import logging -import math -import os -import sys -from dataclasses import dataclass, field -from typing import Optional - -import datasets -from datasets import load_dataset - -import transformers -from transformers import ( - CONFIG_MAPPING, - MODEL_FOR_MASKED_LM_MAPPING, - AutoConfig, - AutoModelForMaskedLM, - AutoTokenizer, - DataCollatorForLanguageModeling, - HfArgumentParser, - Trainer, - TrainingArguments, - set_seed, -) -from transformers.trainer_utils import get_last_checkpoint -from transformers.utils import check_min_version -from transformers.utils.versions import require_version - -# for enable herring -import smdistributed.dataparallel.torch.torch_smddp - -# Will error if the minimal version of Transformers is not installed. Remove at your own risks. -check_min_version("4.10.0") - -require_version( - "datasets>=1.8.0", - "To fix: pip install -r examples/pytorch/language-modeling/requirements.txt", -) - -logger = logging.getLogger(__name__) -MODEL_CONFIG_CLASSES = list(MODEL_FOR_MASKED_LM_MAPPING.keys()) -MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES) - - -@dataclass -class ModelArguments: - """ - Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. - """ - - model_name_or_path: Optional[str] = field( - default=None, - metadata={ - "help": "The model checkpoint for weights initialization." - "Don't set if you want to train a model from scratch." - }, - ) - model_type: Optional[str] = field( - default=None, - metadata={ - "help": "If training from scratch, pass a model type from the list: " - + ", ".join(MODEL_TYPES) - }, - ) - config_overrides: Optional[str] = field( - default=None, - metadata={ - "help": "Override some existing default config settings when a model is trained from scratch. Example: " - "n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index" - }, - ) - config_name: Optional[str] = field( - default=None, - metadata={"help": "Pretrained config name or path if not the same as model_name"}, - ) - tokenizer_name: Optional[str] = field( - default=None, - metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}, - ) - cache_dir: Optional[str] = field( - default=None, - metadata={ - "help": "Where do you want to store the pretrained models downloaded from huggingface.co" - }, - ) - use_fast_tokenizer: bool = field( - default=True, - metadata={ - "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not." - }, - ) - model_revision: str = field( - default="main", - metadata={ - "help": "The specific model version to use (can be a branch name, tag name or commit id)." - }, - ) - use_auth_token: bool = field( - default=False, - metadata={ - "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " - "with private models)." - }, - ) - - def __post_init__(self): - if self.config_overrides is not None and ( - self.config_name is not None or self.model_name_or_path is not None - ): - raise ValueError( - "--config_overrides can't be used in combination with --config_name or --model_name_or_path" - ) - - -@dataclass -class DataTrainingArguments: - """ - Arguments pertaining to what data we are going to input our model for training and eval. - """ - - dataset_name: Optional[str] = field( - default=None, - metadata={"help": "The name of the dataset to use (via the datasets library)."}, - ) - dataset_config_name: Optional[str] = field( - default=None, - metadata={ - "help": "The configuration name of the dataset to use (via the datasets library)." - }, - ) - train_file: Optional[str] = field( - default=None, metadata={"help": "The input training data file (a text file)."} - ) - validation_file: Optional[str] = field( - default=None, - metadata={ - "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)." - }, - ) - overwrite_cache: bool = field( - default=False, - metadata={"help": "Overwrite the cached training and evaluation sets"}, - ) - validation_split_percentage: Optional[int] = field( - default=5, - metadata={ - "help": "The percentage of the train set used as validation set in case there's no validation split" - }, - ) - max_seq_length: Optional[int] = field( - default=None, - metadata={ - "help": "The maximum total input sequence length after tokenization. Sequences longer " - "than this will be truncated." - }, - ) - preprocessing_num_workers: Optional[int] = field( - default=None, - metadata={"help": "The number of processes to use for the preprocessing."}, - ) - mlm_probability: float = field( - default=0.15, - metadata={"help": "Ratio of tokens to mask for masked language modeling loss"}, - ) - line_by_line: bool = field( - default=False, - metadata={ - "help": "Whether distinct lines of text in the dataset are to be handled as distinct sequences." - }, - ) - pad_to_max_length: bool = field( - default=False, - metadata={ - "help": "Whether to pad all samples to `max_seq_length`. " - "If False, will pad the samples dynamically when batching to the maximum length in the batch." - }, - ) - max_train_samples: Optional[int] = field( - default=None, - metadata={ - "help": "For debugging purposes or quicker training, truncate the number of training examples to this " - "value if set." - }, - ) - max_eval_samples: Optional[int] = field( - default=None, - metadata={ - "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this " - "value if set." - }, - ) - - def __post_init__(self): - if self.dataset_name is None and self.train_file is None and self.validation_file is None: - raise ValueError("Need either a dataset name or a training/validation file.") - else: - if self.train_file is not None: - extension = self.train_file.split(".")[-1] - assert extension in [ - "csv", - "json", - "txt", - ], "`train_file` should be a csv, a json or a txt file." - if self.validation_file is not None: - extension = self.validation_file.split(".")[-1] - assert extension in [ - "csv", - "json", - "txt", - ], "`validation_file` should be a csv, a json or a txt file." - - -def main(): - # See all possible arguments in src/transformers/training_args.py - # or by passing the --help flag to this script. - # We now keep distinct sets of args, for a cleaner separation of concerns. - - parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments)) - if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): - # If we pass only one argument to the script and it's the path to a json file, - # let's parse it to get our arguments. - model_args, data_args, training_args = parser.parse_json_file( - json_file=os.path.abspath(sys.argv[1]) - ) - else: - model_args, data_args, training_args = parser.parse_args_into_dataclasses() - - # Setup logging - logging.basicConfig( - format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", - datefmt="%m/%d/%Y %H:%M:%S", - handlers=[logging.StreamHandler(sys.stdout)], - ) - - log_level = training_args.get_process_log_level() - logger.setLevel(log_level) - datasets.utils.logging.set_verbosity(log_level) - transformers.utils.logging.set_verbosity(log_level) - transformers.utils.logging.enable_default_handler() - transformers.utils.logging.enable_explicit_format() - - # Log on each process the small summary: - logger.warning( - f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" - + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" - ) - # Set the verbosity to info of the Transformers logger (on main process only): - logger.info(f"Training/evaluation parameters {training_args}") - - # Detecting last checkpoint. - last_checkpoint = None - if ( - os.path.isdir(training_args.output_dir) - and training_args.do_train - and not training_args.overwrite_output_dir - ): - last_checkpoint = get_last_checkpoint(training_args.output_dir) - if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: - raise ValueError( - f"Output directory ({training_args.output_dir}) already exists and is not empty. " - "Use --overwrite_output_dir to overcome." - ) - elif last_checkpoint is not None and training_args.resume_from_checkpoint is None: - logger.info( - f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change " - "the `--output_dir` or add `--overwrite_output_dir` to train from scratch." - ) - - # Set seed before initializing model. - set_seed(training_args.seed) - - # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) - # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ - # (the dataset will be downloaded automatically from the datasets Hub - # - # For CSV/JSON files, this script will use the column called 'text' or the first column. You can easily tweak this - # behavior (see below) - # - # In distributed training, the load_dataset function guarantee that only one local process can concurrently - # download the dataset. - if data_args.dataset_name is not None: - # Downloading and loading a dataset from the hub. - raw_datasets = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - cache_dir=model_args.cache_dir, - ) - if "validation" not in raw_datasets.keys(): - raw_datasets["validation"] = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - split=f"train[:{data_args.validation_split_percentage}%]", - cache_dir=model_args.cache_dir, - ) - raw_datasets["train"] = load_dataset( - data_args.dataset_name, - data_args.dataset_config_name, - split=f"train[{data_args.validation_split_percentage}%:]", - cache_dir=model_args.cache_dir, - ) - else: - data_files = {} - if data_args.train_file is not None: - data_files["train"] = data_args.train_file - extension = data_args.train_file.split(".")[-1] - if data_args.validation_file is not None: - data_files["validation"] = data_args.validation_file - extension = data_args.validation_file.split(".")[-1] - if extension == "txt": - extension = "text" - raw_datasets = load_dataset( - extension, data_files=data_files, cache_dir=model_args.cache_dir - ) - - # If no validation data is there, validation_split_percentage will be used to divide the dataset. - if "validation" not in raw_datasets.keys(): - raw_datasets["validation"] = load_dataset( - extension, - data_files=data_files, - split=f"train[:{data_args.validation_split_percentage}%]", - cache_dir=model_args.cache_dir, - ) - raw_datasets["train"] = load_dataset( - extension, - data_files=data_files, - split=f"train[{data_args.validation_split_percentage}%:]", - cache_dir=model_args.cache_dir, - ) - - # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at - # https://huggingface.co/docs/datasets/loading_datasets.html. - - # Load pretrained model and tokenizer - # - # Distributed training: - # The .from_pretrained methods guarantee that only one local process can concurrently - # download model & vocab. - config_kwargs = { - "cache_dir": model_args.cache_dir, - "revision": model_args.model_revision, - "use_auth_token": True if model_args.use_auth_token else None, - } - if model_args.config_name: - config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs) - elif model_args.model_name_or_path: - config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs) - else: - config = CONFIG_MAPPING[model_args.model_type]() - logger.warning("You are instantiating a new config instance from scratch.") - if model_args.config_overrides is not None: - logger.info(f"Overriding config: {model_args.config_overrides}") - config.update_from_string(model_args.config_overrides) - - tokenizer_kwargs = { - "cache_dir": model_args.cache_dir, - "use_fast": model_args.use_fast_tokenizer, - "revision": model_args.model_revision, - "use_auth_token": True if model_args.use_auth_token else None, - } - if model_args.tokenizer_name: - tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs) - elif model_args.model_name_or_path: - tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs) - else: - raise ValueError( - "You are instantiating a new tokenizer from scratch. This is not supported by this script." - "You can do it from another script, save it, and load it from here, using --tokenizer_name." - ) - - if model_args.model_name_or_path: - model = AutoModelForMaskedLM.from_pretrained( - model_args.model_name_or_path, - from_tf=bool(".ckpt" in model_args.model_name_or_path), - config=config, - cache_dir=model_args.cache_dir, - revision=model_args.model_revision, - use_auth_token=True if model_args.use_auth_token else None, - ) - else: - logger.info("Training new model from scratch") - model = AutoModelForMaskedLM.from_config(config) - - model.resize_token_embeddings(len(tokenizer)) - - # Preprocessing the datasets. - # First we tokenize all the texts. - if training_args.do_train: - column_names = raw_datasets["train"].column_names - else: - column_names = raw_datasets["validation"].column_names - text_column_name = "text" if "text" in column_names else column_names[0] - - if data_args.max_seq_length is None: - max_seq_length = tokenizer.model_max_length - if max_seq_length > 1024: - logger.warning( - f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " - "Picking 1024 instead. You can change that default value by passing --max_seq_length xxx." - ) - max_seq_length = 1024 - else: - if data_args.max_seq_length > tokenizer.model_max_length: - logger.warning( - f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the" - f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}." - ) - max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length) - - if data_args.line_by_line: - # When using line_by_line, we just tokenize each nonempty line. - padding = "max_length" if data_args.pad_to_max_length else False - - def tokenize_function(examples): - # Remove empty lines - examples[text_column_name] = [ - line for line in examples[text_column_name] if len(line) > 0 and not line.isspace() - ] - return tokenizer( - examples[text_column_name], - padding=padding, - truncation=True, - max_length=max_seq_length, - # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it - # receives the `special_tokens_mask`. - return_special_tokens_mask=True, - ) - - with training_args.main_process_first(desc="dataset map tokenization"): - tokenized_datasets = raw_datasets.map( - tokenize_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=[text_column_name], - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on dataset line_by_line", - ) - else: - # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts. - # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more - # efficient when it receives the `special_tokens_mask`. - def tokenize_function(examples): - return tokenizer(examples[text_column_name], return_special_tokens_mask=True) - - with training_args.main_process_first(desc="dataset map tokenization"): - tokenized_datasets = raw_datasets.map( - tokenize_function, - batched=True, - num_proc=data_args.preprocessing_num_workers, - remove_columns=column_names, - load_from_cache_file=not data_args.overwrite_cache, - desc="Running tokenizer on every text in dataset", - ) - - # Main data processing function that will concatenate all texts from our dataset and generate chunks of - # max_seq_length. - def group_texts(examples): - # Concatenate all texts. - concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()} - total_length = len(concatenated_examples[list(examples.keys())[0]]) - # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can - # customize this part to your needs. - if total_length >= max_seq_length: - total_length = (total_length // max_seq_length) * max_seq_length - # Split by chunks of max_len. - result = { - k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)] - for k, t in concatenated_examples.items() - } - return result - - # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a - # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value - # might be slower to preprocess. - # - # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: - # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map - - with training_args.main_process_first(desc="grouping texts together"): - tokenized_datasets = tokenized_datasets.map( - group_texts, - batched=True, - num_proc=data_args.preprocessing_num_workers, - load_from_cache_file=not data_args.overwrite_cache, - desc=f"Grouping texts in chunks of {max_seq_length}", - ) - - if training_args.do_train: - if "train" not in tokenized_datasets: - raise ValueError("--do_train requires a train dataset") - train_dataset = tokenized_datasets["train"] - if data_args.max_train_samples is not None: - train_dataset = train_dataset.select(range(data_args.max_train_samples)) - - if training_args.do_eval: - if "validation" not in tokenized_datasets: - raise ValueError("--do_eval requires a validation dataset") - eval_dataset = tokenized_datasets["validation"] - if data_args.max_eval_samples is not None: - eval_dataset = eval_dataset.select(range(data_args.max_eval_samples)) - - # Data collator - # This one will take care of randomly masking the tokens. - pad_to_multiple_of_8 = ( - data_args.line_by_line and training_args.fp16 and not data_args.pad_to_max_length - ) - data_collator = DataCollatorForLanguageModeling( - tokenizer=tokenizer, - mlm_probability=data_args.mlm_probability, - pad_to_multiple_of=8 if pad_to_multiple_of_8 else None, - ) - - # Initialize our Trainer - trainer = Trainer( - model=model, - args=training_args, - train_dataset=train_dataset if training_args.do_train else None, - eval_dataset=eval_dataset if training_args.do_eval else None, - tokenizer=tokenizer, - data_collator=data_collator, - ) - - # Training - if training_args.do_train: - checkpoint = None - if training_args.resume_from_checkpoint is not None: - checkpoint = training_args.resume_from_checkpoint - elif last_checkpoint is not None: - checkpoint = last_checkpoint - train_result = trainer.train(resume_from_checkpoint=checkpoint) - trainer.save_model() # Saves the tokenizer too for easy upload - metrics = train_result.metrics - - max_train_samples = ( - data_args.max_train_samples - if data_args.max_train_samples is not None - else len(train_dataset) - ) - metrics["train_samples"] = min(max_train_samples, len(train_dataset)) - - trainer.log_metrics("train", metrics) - trainer.save_metrics("train", metrics) - trainer.save_state() - - # Evaluation - if training_args.do_eval: - logger.info("*** Evaluate ***") - - metrics = trainer.evaluate() - - max_eval_samples = ( - data_args.max_eval_samples - if data_args.max_eval_samples is not None - else len(eval_dataset) - ) - metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset)) - try: - perplexity = math.exp(metrics["eval_loss"]) - except OverflowError: - perplexity = float("inf") - metrics["perplexity"] = perplexity - - trainer.log_metrics("eval", metrics) - trainer.save_metrics("eval", metrics) - - if training_args.push_to_hub: - kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "fill-mask"} - if data_args.dataset_name is not None: - kwargs["dataset_tags"] = data_args.dataset_name - if data_args.dataset_config_name is not None: - kwargs["dataset_args"] = data_args.dataset_config_name - kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}" - else: - kwargs["dataset"] = data_args.dataset_name - - trainer.push_to_hub(**kwargs) - - -def _mp_fn(index): - # For xla_spawn (TPUs) - main() - - -if __name__ == "__main__": - main() From e02536697b43ee37b9c85be0c368b626006a75d4 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Fri, 7 Oct 2022 15:29:38 +0000 Subject: [PATCH 3/9] fix format --- .../language-modeling-multi-gpu-multi-node.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index 31458e6a9f..fb8e2e4962 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -195,13 +195,12 @@ "# For more information about the options, please look into the training scripts\n", "\n", "# Select Instance type for training\n", - "INSTANCE_TYPE = \"ml.p4d.24xlarge\" \n", + "INSTANCE_TYPE = \"ml.p4d.24xlarge\"\n", "NUM_INSTANCES = 2\n", "# Since ml.p4d.24xlarge instance has 8 GPUs, we set num_gpus_per_instance to 8\n", "num_gpus_per_instance = 8\n", "\n", "\n", - "\n", "estimator_args = dict(\n", " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\",\n", " source_dir=\"./scripts\",\n", @@ -224,7 +223,7 @@ " {\"Name\": \"epoch\", \"Regex\": \"'epoch': ([0-9.]*)\"},\n", " {\"Name\": \"train_loss\", \"Regex\": \"'loss': ([0-9.]*)\"},\n", " {\"Name\": \"learning_rate\", \"Regex\": \"'learning_rate': ([0-9.]*)\"},\n", - " ], \n", + " ],\n", ")" ] }, @@ -595,6 +594,7 @@ "outputs": [], "source": [ "import pandas as pd\n", + "\n", "pd.DataFrame([n[\"summary\"], o[\"summary\"]], index=[\"Native\", \"Compiler-enhanced\"])" ] }, From b2254e6f298130327321abd466241598f60e6bb6 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Thu, 13 Oct 2022 20:57:14 +0000 Subject: [PATCH 4/9] edit by comments and update learning rate --- ...nguage-modeling-multi-gpu-multi-node.ipynb | 793 +++++++++++++++--- 1 file changed, 660 insertions(+), 133 deletions(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index fb8e2e4962..68b71253d2 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "3c17b89c", + "id": "98f52af1", "metadata": {}, "source": [ "# Compile and Train the GPT2 Model using the Transformers Trainer API with the SST2 Dataset for Multi-Node Multi-GPU Training" @@ -10,22 +10,22 @@ }, { "cell_type": "markdown", - "id": "297eba0a", + "id": "84e26b8a", "metadata": {}, "source": [ "1. [Introduction](#Introduction) \n", - "2. [Development Environment and Permissions](#Development-Environment-and-Permissions)\n", + "2. [Development Environment](#Development-Environment)\n", " 1. [Installation](#Installation) \n", - " 2. [Permissions](#Permissions)\n", + " 2. [SageMaker Environment](#SageMaker-Environment)\n", "3. [SageMaker Training Job](#SageMaker-Training-Job) \n", - " 1. [Training with Native PyTorch](#Training-with-Native-PyTorch) \n", - " 2. [Training with Optimized PyTorch](#Training-with-Optimized-PyTorch) \n", - " 3. [Analysis](#Analysis) " + " 1. [Training with Native PyTorch + SM DDP](#Training-with-Native-PyTorch-+-SM-DDP) \n", + " 2. [Training with SageMaker Training Compiler](#Training-with-SageMaker-Training-Compiler) \n", + "4. [Analysis](#Analysis) " ] }, { "cell_type": "markdown", - "id": "32474760", + "id": "392d66aa", "metadata": {}, "source": [ "## SageMaker Training Compiler Overview\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "a76ece43", + "id": "be898d9d", "metadata": {}, "source": [ "## Development Environment " @@ -55,40 +55,347 @@ }, { "cell_type": "markdown", - "id": "62fe1493", + "id": "6346bd12", + "metadata": {}, + "source": [ + "### Installation" + ] + }, + { + "cell_type": "markdown", + "id": "d49ec5c5", "metadata": {}, "source": [ - "### Installation\n", - "\n", "This example notebook requires the **SageMaker Python SDK v2.108.0** and **transformers v4.21**." ] }, { "cell_type": "code", - "execution_count": null, - "id": "1a3240eb", + "execution_count": 1, + "id": "4e4b43d1", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n", + "Requirement already satisfied: sagemaker>=2.108.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (2.112.2)\n", + "Requirement already satisfied: botocore in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.27.89)\n", + "Collecting botocore\n", + " Downloading botocore-1.27.90-py3-none-any.whl (9.2 MB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.2/9.2 MB\u001b[0m \u001b[31m49.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: boto3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.24.89)\n", + "Collecting boto3\n", + " Downloading boto3-1.24.90-py3-none-any.whl (132 kB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.5/132.5 KB\u001b[0m \u001b[31m32.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: awscli in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.25.90)\n", + "Collecting awscli\n", + " Downloading awscli-1.25.91-py3-none-any.whl (3.9 MB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.9/3.9 MB\u001b[0m \u001b[31m83.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: s3fs in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (0.4.2)\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.8.2-py3-none-any.whl (27 kB)\n", + "Requirement already satisfied: typing-extensions in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (4.4.0)\n", + "Requirement already satisfied: torch==1.11.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.11.0)\n", + "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.5.0)\n", + "Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.23.4)\n", + "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.1.5)\n", + "Requirement already satisfied: attrs<23,>=20.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (21.2.0)\n", + "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (21.3)\n", + "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (1.0.1)\n", + "Requirement already satisfied: schema in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.7.5)\n", + "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (4.8.2)\n", + "Requirement already satisfied: protobuf<4.0,>=3.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (3.20.1)\n", + "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.2.0)\n", + "Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.2.8)\n", + "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (1.26.8)\n", + "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (0.10.0)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (2.8.2)\n", + "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from boto3) (0.6.0)\n", + "Requirement already satisfied: PyYAML<5.5,>=3.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (5.4.1)\n", + "Requirement already satisfied: rsa<4.8,>=3.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (4.7.2)\n", + "Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (0.4.3)\n", + "Requirement already satisfied: docutils<0.17,>=0.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (0.15.2)\n", + "Collecting aiobotocore~=2.4.0\n", + " Using cached aiobotocore-2.4.0-py3-none-any.whl (65 kB)\n", + "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (3.8.1)\n", + "Collecting fsspec==2022.8.2\n", + " Using cached fsspec-2022.8.2-py3-none-any.whl (140 kB)\n", + "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas) (2021.3)\n", + "Requirement already satisfied: aioitertools>=0.5.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiobotocore~=2.4.0->s3fs) (0.8.0)\n", + "INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.\n", + "Collecting numpy\n", + " Using cached numpy-1.23.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)\n", + "INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.\n", + "Collecting pandas\n", + " Using cached pandas-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)\n", + "INFO: pip is looking at multiple versions of typing-extensions to determine which version is compatible with other requirements. This could take a while.\n", + "Collecting typing-extensions\n", + " Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)\n", + "INFO: pip is looking at multiple versions of fsspec to determine which version is compatible with other requirements. This could take a while.\n", + "INFO: pip is looking at multiple versions of s3fs to determine which version is compatible with other requirements. This could take a while.\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.8.1-py3-none-any.whl (27 kB)\n", + "Collecting fsspec==2022.8.1\n", + " Using cached fsspec-2022.8.1-py3-none-any.whl (140 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.8.0-py3-none-any.whl (27 kB)\n", + "Collecting fsspec==2022.8.0\n", + " Using cached fsspec-2022.8.0-py3-none-any.whl (140 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.7.1-py3-none-any.whl (27 kB)\n", + "Collecting fsspec==2022.7.1\n", + " Using cached fsspec-2022.7.1-py3-none-any.whl (141 kB)\n", + "Collecting aiobotocore~=2.3.4\n", + " Using cached aiobotocore-2.3.4-py3-none-any.whl (64 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.7.0-py3-none-any.whl (27 kB)\n", + "Collecting fsspec==2022.7.0\n", + " Using cached fsspec-2022.7.0-py3-none-any.whl (141 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.5.0-py3-none-any.whl (27 kB)\n", + "Collecting fsspec==2022.5.0\n", + " Using cached fsspec-2022.5.0-py3-none-any.whl (140 kB)\n", + "Requirement already satisfied: aiobotocore~=2.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (2.3.3)\n", + "Collecting aiobotocore~=2.3.0\n", + " Using cached aiobotocore-2.3.2.tar.gz (104 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-2.3.1.tar.gz (65 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-2.3.0.tar.gz (65 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting s3fs\n", + " Using cached s3fs-2022.3.0-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2022.3.0\n", + " Using cached fsspec-2022.3.0-py3-none-any.whl (136 kB)\n", + "Collecting aiobotocore~=2.2.0\n", + " Using cached aiobotocore-2.2.0.tar.gz (59 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting s3fs\n", + " Using cached s3fs-2022.2.0-py3-none-any.whl (26 kB)\n", + "Collecting aiobotocore~=2.1.0\n", + " Using cached aiobotocore-2.1.2.tar.gz (58 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting fsspec==2022.02.0\n", + " Using cached fsspec-2022.2.0-py3-none-any.whl (134 kB)\n", + "Collecting aiobotocore~=2.1.0\n", + " Using cached aiobotocore-2.1.1.tar.gz (57 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-2.1.0.tar.gz (54 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hINFO: pip is looking at multiple versions of fsspec to determine which version is compatible with other requirements. This could take a while.\n", + "INFO: pip is looking at multiple versions of s3fs to determine which version is compatible with other requirements. This could take a while.\n", + "Collecting s3fs\n", + " Using cached s3fs-2022.1.0-py3-none-any.whl (25 kB)\n", + "Collecting fsspec==2022.01.0\n", + " Using cached fsspec-2022.1.0-py3-none-any.whl (133 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.11.1-py3-none-any.whl (25 kB)\n", + "Requirement already satisfied: fsspec==2021.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (2021.11.1)\n", + "Collecting aiobotocore~=2.0.1\n", + " Using cached aiobotocore-2.0.1-py3-none-any.whl\n", + "Requirement already satisfied: wrapt>=1.10.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiobotocore~=2.0.1->s3fs) (1.13.3)\n", + "Collecting fsspec==2021.11.1\n", + " Using cached fsspec-2021.11.1-py3-none-any.whl (132 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.11.0-py3-none-any.whl (25 kB)\n", + "Collecting aiobotocore~=1.4.1\n", + " Using cached aiobotocore-1.4.2.tar.gz (52 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting fsspec==2021.11.0\n", + " Using cached fsspec-2021.11.0-py3-none-any.whl (132 kB)\n", + "Collecting aiobotocore~=1.4.1\n", + " Using cached aiobotocore-1.4.1.tar.gz (52 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting s3fs\n", + " Using cached s3fs-2021.10.1-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2021.10.1\n", + " Using cached fsspec-2021.10.1-py3-none-any.whl (125 kB)\n", + "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.10.0-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2021.10.0\n", + " Using cached fsspec-2021.10.0-py3-none-any.whl (125 kB)\n", + "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.9.0-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2021.09.0\n", + " Using cached fsspec-2021.9.0-py3-none-any.whl (123 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.8.1-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2021.08.1\n", + " Using cached fsspec-2021.8.1-py3-none-any.whl (119 kB)\n", + "Collecting aiobotocore~=1.4.0\n", + " Using cached aiobotocore-1.4.0.tar.gz (51 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25hCollecting s3fs\n", + " Using cached s3fs-2021.8.0-py3-none-any.whl (26 kB)\n", + "Collecting fsspec==2021.07.0\n", + " Using cached fsspec-2021.7.0-py3-none-any.whl (118 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.7.0-py3-none-any.whl (25 kB)\n", + "Collecting aiobotocore>=1.0.1\n", + " Using cached aiobotocore-2.0.0.tar.gz (52 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.3.3.tar.gz (50 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.3.2.tar.gz (49 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.3.1.tar.gz (48 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.3.0.tar.gz (48 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.2.2.tar.gz (48 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.2.1.tar.gz (48 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.2.0.tar.gz (47 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25ldone\n", + "\u001b[?25h Using cached aiobotocore-1.1.2-py3-none-any.whl (45 kB)\n", + " Using cached aiobotocore-1.1.1-py3-none-any.whl (45 kB)\n", + " Using cached aiobotocore-1.1.0-py3-none-any.whl (43 kB)\n", + " Using cached aiobotocore-1.0.7-py3-none-any.whl (42 kB)\n", + " Using cached aiobotocore-1.0.6-py3-none-any.whl (42 kB)\n", + " Using cached aiobotocore-1.0.5-py3-none-any.whl (42 kB)\n", + " Using cached aiobotocore-1.0.4-py3-none-any.whl (41 kB)\n", + " Using cached aiobotocore-1.0.3-py3-none-any.whl (40 kB)\n", + " Using cached aiobotocore-1.0.2-py3-none-any.whl (40 kB)\n", + " Using cached aiobotocore-1.0.1-py3-none-any.whl (40 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.6.1-py3-none-any.whl (25 kB)\n", + "Collecting fsspec==2021.06.1\n", + " Using cached fsspec-2021.6.1-py3-none-any.whl (115 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.6.0-py3-none-any.whl (24 kB)\n", + "Collecting fsspec==2021.06.0\n", + " Using cached fsspec-2021.6.0-py3-none-any.whl (114 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.5.0-py3-none-any.whl (24 kB)\n", + "Collecting fsspec==2021.05.0\n", + " Using cached fsspec-2021.5.0-py3-none-any.whl (111 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-2021.4.0-py3-none-any.whl (23 kB)\n", + "Collecting fsspec==2021.04.0\n", + " Using cached fsspec-2021.4.0-py3-none-any.whl (108 kB)\n", + "Collecting s3fs\n", + " Using cached s3fs-0.6.0-py3-none-any.whl (23 kB)\n", + " Using cached s3fs-0.5.2-py3-none-any.whl (22 kB)\n", + " Using cached s3fs-0.5.1-py3-none-any.whl (21 kB)\n", + " Using cached s3fs-0.5.0-py3-none-any.whl (21 kB)\n", + "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker>=2.108.0) (3.6.0)\n", + "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->sagemaker>=2.108.0) (3.0.6)\n", + "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker>=2.108.0) (1.16.0)\n", + "Requirement already satisfied: pyasn1>=0.1.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from rsa<4.8,>=3.1.2->awscli) (0.4.8)\n", + "Requirement already satisfied: pox>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.3.0)\n", + "Requirement already satisfied: multiprocess>=0.70.12 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.70.12.2)\n", + "Requirement already satisfied: dill>=0.3.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.3.4)\n", + "Requirement already satisfied: ppft>=1.6.6.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (1.6.6.4)\n", + "Requirement already satisfied: contextlib2>=0.5.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from schema->sagemaker>=2.108.0) (21.6.0)\n", + "Installing collected packages: botocore, boto3, awscli\n", + " Attempting uninstall: botocore\n", + " Found existing installation: botocore 1.27.89\n", + " Uninstalling botocore-1.27.89:\n", + " Successfully uninstalled botocore-1.27.89\n", + " Attempting uninstall: boto3\n", + " Found existing installation: boto3 1.24.89\n", + " Uninstalling boto3-1.24.89:\n", + " Successfully uninstalled boto3-1.24.89\n", + " Attempting uninstall: awscli\n", + " Found existing installation: awscli 1.25.90\n", + " Uninstalling awscli-1.25.90:\n", + " Successfully uninstalled awscli-1.25.90\n", + "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "aiobotocore 2.3.3 requires botocore<1.24.22,>=1.24.21, but you have botocore 1.27.90 which is incompatible.\u001b[0m\u001b[31m\n", + "\u001b[0mSuccessfully installed awscli-1.25.91 boto3-1.24.90 botocore-1.27.90\n", + "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.\n", + "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n", + "\u001b[0m" + ] + } + ], "source": [ "!pip install \"sagemaker>=2.108.0\" botocore boto3 awscli s3fs typing-extensions \"torch==1.11.0\" pandas numpy --upgrade" ] }, { "cell_type": "code", - "execution_count": null, - "id": "a2b9276e", + "execution_count": 2, + "id": "0c3adc03", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n", + "Requirement already satisfied: transformers==4.21 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (4.21.0)\n", + "Requirement already satisfied: datasets in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (2.6.0)\n", + "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (2021.11.10)\n", + "Requirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (5.4.1)\n", + "Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (2.26.0)\n", + "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (21.3)\n", + "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (3.4.0)\n", + "Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (0.12.1)\n", + "Requirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (4.62.3)\n", + "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (0.9.0)\n", + "Requirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (1.23.4)\n", + "Requirement already satisfied: dill<0.3.6 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.3.4)\n", + "Requirement already satisfied: xxhash in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (3.0.0)\n", + "Requirement already satisfied: aiohttp in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (3.8.1)\n", + "Requirement already satisfied: fsspec[http]>=2021.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (2021.11.1)\n", + "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (1.5.0)\n", + "Requirement already satisfied: pyarrow>=6.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (7.0.0)\n", + "Requirement already satisfied: multiprocess in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.70.12.2)\n", + "Requirement already satisfied: responses<0.19 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.18.0)\n", + "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers==4.21) (4.4.0)\n", + "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->transformers==4.21) (3.0.6)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (2021.10.8)\n", + "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (1.26.8)\n", + "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (3.1)\n", + "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (2.0.7)\n", + "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.7.2)\n", + "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (4.0.1)\n", + "Requirement already satisfied: frozenlist>=1.1.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.2.0)\n", + "Requirement already satisfied: attrs>=17.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (21.2.0)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (5.2.0)\n", + "Requirement already satisfied: aiosignal>=1.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.2.0)\n", + "Requirement already satisfied: python-dateutil>=2.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas->datasets) (2.8.2)\n", + "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas->datasets) (2021.3)\n", + "Requirement already satisfied: six>=1.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n", + "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.\n", + "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n", + "\u001b[0m" + ] + } + ], "source": [ "!pip install \"transformers==4.21\" datasets --upgrade" ] }, { "cell_type": "code", - "execution_count": null, - "id": "824bc2b5", + "execution_count": 3, + "id": "4c979dbd", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4\n", + " warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sagemaker: 2.112.2\n", + "transformers: 4.21.0\n" + ] + } + ], "source": [ "import botocore\n", "import boto3\n", @@ -102,7 +409,7 @@ }, { "cell_type": "markdown", - "id": "e97e9356", + "id": "52fad7c5", "metadata": {}, "source": [ "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for datasets library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", @@ -118,18 +425,28 @@ }, { "cell_type": "markdown", - "id": "201cc329", + "id": "710cf271", "metadata": {}, "source": [ - "### SageMaker environment " + "### SageMaker Environment " ] }, { "cell_type": "code", - "execution_count": null, - "id": "76b34d36", + "execution_count": 4, + "id": "adbc14fb", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sagemaker role arn: arn:aws:iam::875423407011:role/AdminRole\n", + "sagemaker bucket: sagemaker-us-west-2-875423407011\n", + "sagemaker session region: us-west-2\n" + ] + } + ], "source": [ "import sagemaker\n", "\n", @@ -152,36 +469,26 @@ }, { "cell_type": "markdown", - "id": "06e57ac1", + "id": "b021041c", "metadata": {}, "source": [ "## SageMaker Training Job\n", "\n", "To create a SageMaker training job, we use a HuggingFace/PyTorch estimator. Using the estimator, you can define which training script should SageMaker use through entry_point, which instance_type to use for training, which hyperparameters to pass, and so on.\n", "\n", - "When a SageMaker training job starts, SageMaker takes care of starting and managing all the required machine learning instances, picks up the HuggingFace Deep Learning Container, uploads your training script, and downloads the data from sagemaker_session_bucket into the container at /opt/ml/input/data.\n", - "\n", - "In the following section, you learn how to set up two versions of the SageMaker HuggingFace/PyTorch estimator, a native one without the compiler and an optimized one with the compiler." - ] - }, - { - "cell_type": "markdown", - "id": "0b5f2232", - "metadata": {}, - "source": [ - "### Training Setup" + "When a SageMaker training job starts, SageMaker takes care of starting and managing all the required machine learning instances, picks up the HuggingFace Deep Learning Container, uploads your training script, and downloads the data from sagemaker_session_bucket into the container at /opt/ml/input/data." ] }, { "cell_type": "code", - "execution_count": null, - "id": "7f1b2444", + "execution_count": 5, + "id": "c874d8a8", "metadata": {}, "outputs": [], "source": [ "# Here we configure the training job. Please configure the appropriate options below:\n", "\n", - "EPOCHS = 600\n", + "EPOCHS = 1000\n", "\n", "# Choose between Causal Language Model and Masked Language Model\n", "LANGUAGE_MODELING_LOSS = \"clm\" # or \"mlm\"\n", @@ -198,9 +505,26 @@ "INSTANCE_TYPE = \"ml.p4d.24xlarge\"\n", "NUM_INSTANCES = 2\n", "# Since ml.p4d.24xlarge instance has 8 GPUs, we set num_gpus_per_instance to 8\n", - "num_gpus_per_instance = 8\n", - "\n", + "num_gpus_per_instance = 8" + ] + }, + { + "cell_type": "markdown", + "id": "6431fe3c", + "metadata": {}, + "source": [ + "First, we define some basic parameters common to all estimators.\n", "\n", + "**Note**: We recommend you to turn the SageMaker Debugger's profiling and debugging tools off to avoid additional overheads." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "5ee306d6", + "metadata": {}, + "outputs": [], + "source": [ "estimator_args = dict(\n", " entry_point=f\"run_{LANGUAGE_MODELING_LOSS}.py\",\n", " source_dir=\"./scripts\",\n", @@ -229,29 +553,19 @@ }, { "cell_type": "markdown", - "id": "ac0f0182", - "metadata": {}, - "source": [ - "### Training with SageMaker Distributed Data Parallel " - ] - }, - { - "cell_type": "markdown", - "id": "3dbfd6a1", + "id": "d03ac353", "metadata": {}, "source": [ - "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training." + "Next, we define some basic arguments to be passed to the training script." ] }, { "cell_type": "code", - "execution_count": null, - "id": "187569b9", + "execution_count": 7, + "id": "33960dc2", "metadata": {}, "outputs": [], "source": [ - "from sagemaker.pytorch import PyTorch\n", - "\n", "# hyperparameters are passed to the training entrypoint as arguments\n", "hyperparameters = {\n", " MODEL_CONFIG: MODEL_NAME,\n", @@ -273,20 +587,55 @@ "}" ] }, + { + "cell_type": "markdown", + "id": "e00a9431", + "metadata": {}, + "source": [ + "In the following sections, we will create estimators and start training." + ] + }, + { + "cell_type": "markdown", + "id": "42a0e4e7", + "metadata": {}, + "source": [ + "### Training with Native PyTorch + SM DDP" + ] + }, + { + "cell_type": "markdown", + "id": "32039258", + "metadata": {}, + "source": [ + "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training.\n", + "\n", + "This example uses HuggingFace training script run_clm.py, which you can find it inside the scripts folder." + ] + }, { "cell_type": "code", - "execution_count": null, - "id": "f4afd4ae", + "execution_count": 8, + "id": "98391f00", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "'trcomp-pt-example-2022-10-13-20-02-06-586'" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ + "from sagemaker.pytorch import PyTorch\n", + "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", - " float(\"5e-5\")\n", - " / 32\n", - " * hyperparameters[\"per_device_train_batch_size\"]\n", - " * num_gpus_per_instance\n", - " * NUM_INSTANCES\n", + " float(\"5e-5\") / 32 * hyperparameters[\"per_device_train_batch_size\"]\n", ")\n", "\n", "# configure the training job\n", @@ -296,7 +645,7 @@ " hyperparameters=hyperparameters,\n", " distribution={\n", " \"smdistributed\": {\"dataparallel\": {\"enabled\": True}}\n", - " }, # Use SageMaker Data Distributed Parallel to train across nodes/GPUs.\n", + " }, # Use SageMaker Distributed Data Parallel to train across nodes/GPUs.\n", ")\n", "\n", "# Start the training job\n", @@ -306,7 +655,7 @@ }, { "cell_type": "markdown", - "id": "6a9388b4", + "id": "64ff8e38", "metadata": {}, "source": [ "### Training with SageMaker Training Compiler " @@ -314,20 +663,29 @@ }, { "cell_type": "markdown", - "id": "5226190d", + "id": "10108936", "metadata": {}, "source": [ - "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately.\n", - "\n", - "**Note:** We recommend you to turn the SageMaker Debugger's profiling and debugging tools off when you use compilation to avoid additional overheads." + "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately." ] }, { "cell_type": "code", - "execution_count": null, - "id": "403e0e24", + "execution_count": 9, + "id": "afd6d9d7", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "'trcomp-pt-example-2022-10-13-20-02-07-185'" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", "\n", @@ -336,11 +694,7 @@ "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", - " float(\"5e-5\")\n", - " / 32\n", - " * hyperparameters[\"per_device_train_batch_size\"]\n", - " * num_gpus_per_instance\n", - " * NUM_INSTANCES\n", + " float(\"5e-5\") / 32 * hyperparameters[\"per_device_train_batch_size\"]\n", ")\n", "\n", "# configure the training job\n", @@ -360,7 +714,7 @@ }, { "cell_type": "markdown", - "id": "63403eee", + "id": "8c95d3f9", "metadata": {}, "source": [ "### Wait for training jobs to complete\n" @@ -368,8 +722,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "cd5455f8", + "execution_count": 23, + "id": "f6a9d068", "metadata": {}, "outputs": [], "source": [ @@ -385,7 +739,7 @@ }, { "cell_type": "markdown", - "id": "13ca3e97", + "id": "f08e5948", "metadata": {}, "source": [ "## Analysis" @@ -393,7 +747,7 @@ }, { "cell_type": "markdown", - "id": "b05fb71b", + "id": "ab81168f", "metadata": {}, "source": [ "**Note:** If the estimator object is no longer available due to a kernel break or refresh, you need to directly use the training job name and manually attach the training job to a new HuggingFace estimator. For example:\n", @@ -405,7 +759,7 @@ }, { "cell_type": "markdown", - "id": "69097726", + "id": "a54ac611", "metadata": {}, "source": [ "### Load logs of the training job *with* SageMaker Training Compiler" @@ -413,8 +767,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "59e6ebf2", + "execution_count": 24, + "id": "26da5643", "metadata": {}, "outputs": [], "source": [ @@ -426,7 +780,7 @@ }, { "cell_type": "markdown", - "id": "638edd10", + "id": "8245afb8", "metadata": {}, "source": [ "### Load logs of the training job *without* SageMaker Training Compiler" @@ -434,8 +788,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "4d87d7e4", + "execution_count": 25, + "id": "f57c7a57", "metadata": {}, "outputs": [], "source": [ @@ -447,7 +801,7 @@ }, { "cell_type": "markdown", - "id": "69f23ac1", + "id": "1dbc9558", "metadata": {}, "source": [ "### Create helper functions for analysis" @@ -455,8 +809,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "e863252a", + "execution_count": 26, + "id": "79b199de", "metadata": {}, "outputs": [], "source": [ @@ -497,7 +851,7 @@ }, { "cell_type": "markdown", - "id": "901c0346", + "id": "65ceecc4", "metadata": {}, "source": [ "### Plot Optimized vs Native Training Throughput\n", @@ -507,8 +861,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "1c053bbe", + "execution_count": 27, + "id": "2382157d", "metadata": { "scrolled": true }, @@ -528,26 +882,51 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "21eea555", + "execution_count": 28, + "id": "3d71417c", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "([,\n", + " ],\n", + " [Text(1.0, 0, 'PT + SM DDP'), Text(1.5, 0, 'PT + Compiler')])" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAElCAYAAAAV9s4VAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkbElEQVR4nO3deZhcZZXH8e+PsGgQBEzLkgSCEFRgMEBkccGgDgZhBGcUiIyAgwZGGHXGUQEZQCSyKOIgDMgaUFYFBDHKJgIuEZoQSVgCYTMJAZo1bAYSzvzxvgU3RVXfqqSrqzr9+zxPPXXve7dzO5U69d7lXEUEZmZmvVmh3QGYmVnnc7IwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYQOepN9I2rev5+1Lkh6W9PH+3m4tkkLSxu2OwwaWFdsdgA1Okl4ojA4FFgKL8/gBEXFBo+uKiJ1bMW+jJO0N/CSPDgFWAV4qbPNtfb3NTiRpFPAQsFJELGpzONbH3LOwtoiIt1VewN+Afyq0vZ4oJHX8D5qIuKCwLzsDj1btX1MGwj7b4ONkYR1F0jhJcyV9S9JjwLmS1pR0taQeSc/k4RGFZX4v6Yt5eD9Jf5D0gzzvQ5J2Xsp5N5R0s6TnJV0v6VRJP1uG3Rsj6U5Jz0m6RNJbetnnVST9SNKj+fUjSasU4676u71+aEnSOyT9StICSbdJOqZ6fuDjku6X9GzeLxXW/UdJp+Q475X0scJ2ljicJumowt/k5vz+rKQXJG2/DH8r6zBOFtaJ1gHWAjYAJpI+p+fm8fWBl4FTell+W2AWMAw4ATi78mXY5LwXArcC7wCOAj6/1HuU7AGMBzYEtgD2K0yr3udvA9sBY4D3AdsAhze4nVOBF/M6982varsC789x7AF8ojBtW+AB0t/kSOBySWs1sN0d8vsauVf15wbjtQHAycI60WvAkRGxMCJejoinIuKyiHgpIp4HJgEf6WX5RyLizIhYDJwHrAus3cy8ktYnfZkeERGvRMQfgKuWcb9OjohHI+Jp4FekRFCxxD4DewNHR8QTEdEDfIcGkpWkIcC/5HW9FBF35/2qdlxEPBsRfwNurIrlCeBHEfFqRFxCSqa7NLuztnxxsrBO1BMRf6+MSBoq6SeSHpG0gHS4Y438xVjLY5WBiKicaK537qDevOsBTxfaAOY0uR91t0U6AV6MaYl9ztt/pDD+SG4r00W6cKUYa624e4tlXixZYbTRbdtyzMnCOlF1KeSvA+8Gto2I1XnjcEe9Q0t9YT6wlqShhbaRLdxe9T4/SjokVbF+boN0iOn1uCStU5ivB1gEjCi0NRv38KrDdnW3TTrUVeES1ssxJwsbCFYjnad4Nh87P7LVG4yIR4Bu4ChJK+eTtf/U6u0WXAQcLqlL0jDgCKByIvmvwGaSxuST5EcV4l4MXJ7jHirpPcA+TW77ncBXJK0k6bPAe4Epedp0YK88bSzwmcJyPaTDae9qcns2ADhZ2EDwI+CtwJPAVOC3/bTdvYHtgaeAY4BLSPeD9IdjSMnqTmAGMC23ERH3AUcD1wP3A9VXOh0MvJ10qOmnpMTTTNx/AUaT/t6TgM9ExFN52v8AGwHPkM6jXFhZKB+ymwT8MV9ltV0T27QOJz/8yKwxki4B7o2Ilvds+pKk44F1IqL0znVJ+wFfjIgPtTwwG1DcszCrQ9L7JW0kaQVJ44HdgF+2OaxSkt4jaQsl2wD7A1e0Oy4b2HynqFl965CO/78DmAv8e0Tc0d6QGrIa6dDTesDjwInAlW2NyAY8H4YyM7NSPgxlZmalnCxswJF0kaTdm5h/b0nXNjjvm+ou9SVJh0k6aymW65gS5wC5dtW9krraHYv1DycLG1AkbUGqlXRlHq/55V78cs1VYXfq30hri4jvRcQX+3ObNYr/jcqFB5f6nGVELATOAQ7pixit8zlZ2EBzAHBBDICTbcvyZdzJCvt1IbBvpRquLd+cLGyg2Rm4qZkFqnsfknaSNCuX4P4/STcply0vzFOvbPnbJZ0tab6kebn895DCdv4o6SRJT1G4s7qw/OslvSW9RdLPJD2Vb2K7TVK9gocA75d0d47r3Hz3dmW9u0qantfzp9wDQ9JPSeU6fpXLhn+TOqXEJf2bpHvy+q+RtEFh/SHpIEn3k24EJCLmkm7O8813g4CThQ0YklYllfeetQzrGAb8AjiUdEnsLOADVbP1VrZ8Mqn20sbAlsBOwBerln2QVOV2Ukk4+5LutB6ZYzmQVNaknr1JpcQ3AjYhlyyXtCXpkNABeT0/Aa6StEpEfJ4lHy51AjVKiUvaDTgM+GdSMcJbSJffFu2e92/TQts9pMOCtpxzsrCBZI38/nxV+3b5F/XrL9Kv6Vo+CdwVEZfnR3+ezJIVWKF+2fK18/Jfi4gXI+IJ4CRgr8Kyj0bEjyNiUS413ptXSV/uG0fE4oi4PSIW9DL/KRExJ5c4nwRMyO0TgZ9ExF/yes4jlfdo5hf/gcCxEXFP/rt8j/SwpmIxw2Mj4umq/XqeN/5dbDnmZGEDybP5fbWq9qkRsUbxRfo1Xct6FEp253Mfc6vmqVe2fANgJWB+ISn9hFR4r6KZMuY/Ba4BLlZ6Gt4JklbqZf7iuotlwzcAvl6VLEfSXFnxDYD/LSz/NKmq7/A6269YjTf+XWw55mRhA0ZEvEh6gtsmy7Ca+RTKd+fDSyPqz76EOaRf7MMKiWn1iNisGGajgeSHC30nIjYlHQrbld4rxBZLjRfLhs8BJlUlzKERUTmMVB1TrRjnAAdUreOtEfGnkuXeS6qCa8s5JwsbaKbQ+1Pyyvwa+AdJu+ereg5iyWcy1BUR84FrgRMlrZ5rRm0kaanikbSjpH/IJ8gXkA5LvdbLIgdJGqFUpv3bpCq4AGcCB0raNteDWlXSLpIqPbDHWbJseK1S4qcDh0raLMf2dqXy5L3FP5z0KNipDe2wDWhOFjbQnAHsXTjh3JSIeBL4LOnE9VOkk7XdNF7Cex9gZeBu0pVAvyCd01ga6+TlF5BOFN9EOjRVz4WkZPUgqYdVKVneDXyJ9FzyZ4DZLPl872NJz8Z4VtJ/1yolHhFXAMeTDoktAGaSrjzrzeeA8/I9F7acc20oG3AkXQhcGhG/7IN1rUA6Z7F3RNy4rOsbLPK9FX8Fdsgn+m0552Rhg46kT5Ae8PMy8A3Soah3NXD1ktmg5cNQNhhtTzqM8yTpUam7O1GY9c49CzMzK+WehZmZlVouC50BDBs2LEaNGtXuMMzMBozbb7/9yYioWXZ+uU0Wo0aNoru7u91hmJkNGJIeqTfNh6HMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKzUcnsHt5kBR7293RFYfzvquZas1j0LMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZqZYlC0nnSHpC0sxC2yWSpufXw5Km5/ZRkl4uTDu9sMzWkmZImi3pZElqVcxmZlZbK+/gngycApxfaYiIPSvDkk4EircaPhARY2qs5zTgS8BfgCnAeOA3fR+umZnV07KeRUTcDDxda1ruHewBXNTbOiStC6weEVMjIkiJZ/c+DtXMzEq065zFh4HHI+L+QtuGku6QdJOkD+e24cDcwjxzc1tNkiZK6pbU3dPT0/dRm5kNUu1KFhNYslcxH1g/IrYE/gu4UNLqza40Is6IiLERMbarq6uPQjUzs36vOitpReCfga0rbRGxEFiYh2+X9ACwCTAPGFFYfERuMzOzftSOnsXHgXsj4vXDS5K6JA3Jw+8CRgMPRsR8YIGk7fJ5jn2AK9sQs5nZoNbKS2cvAv4MvFvSXEn750l78eYT2zsAd+ZLaX8BHBgRlZPjXwbOAmYDD+AroczM+l3LDkNFxIQ67fvVaLsMuKzO/N3A5n0anJmZNcV3cJuZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpVqWLCSdI+kJSTMLbUdJmidpen59sjDtUEmzJc2S9IlC+/jcNlvSIa2K18zM6mtlz2IyML5G+0kRMSa/pgBI2hTYC9gsL/N/koZIGgKcCuwMbApMyPOamVk/WrFVK46ImyWNanD23YCLI2Ih8JCk2cA2edrsiHgQQNLFed67+zpeMzOrrx3nLA6WdGc+TLVmbhsOzCnMMze31WuvSdJESd2Sunt6evo6bjOzQau/k8VpwEbAGGA+cGJfrjwizoiIsRExtqurqy9XbWY2qLXsMFQtEfF4ZVjSmcDVeXQeMLIw64jcRi/tZmbWT/q1ZyFp3cLop4HKlVJXAXtJWkXShsBo4FbgNmC0pA0lrUw6CX5Vf8ZsZmYt7FlIuggYBwyTNBc4EhgnaQwQwMPAAQARcZekS0knrhcBB0XE4ryeg4FrgCHAORFxV6tiNjOz2lp5NdSEGs1n9zL/JGBSjfYpwJQ+DM3MzJrkO7jNzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1ItSxaSzpH0hKSZhbbvS7pX0p2SrpC0Rm4fJellSdPz6/TCMltLmiFptqSTJalVMZuZWW2t7FlMBsZXtV0HbB4RWwD3AYcWpj0QEWPy68BC+2nAl4DR+VW9TjMza7GWJYuIuBl4uqrt2ohYlEenAiN6W4ekdYHVI2JqRARwPrB7C8I1M7NetPOcxb8BvymMbyjpDkk3SfpwbhsOzC3MMze31SRpoqRuSd09PT19H7GZ2SDVULKQtKqkFQrjK0gaurQblfRtYBFwQW6aD6wfEVsC/wVcKGn1ZtcbEWdExNiIGNvV1bW04ZmZWZVGexY3AMXkMBS4fmk2KGk/YFdg73xoiYhYGBFP5eHbgQeATYB5LHmoakRuMzOzftRosnhLRLxQGcnDTfcsJI0Hvgl8KiJeKrR3SRqSh99FOpH9YETMBxZI2i5fBbUPcGWz2zUzs2WzYoPzvShpq4iYBulyVuDl3haQdBEwDhgmaS5wJOnqp1WA6/IVsFPzlU87AEdLehV4DTgwIionx79MurLqraRzHMXzHC0x6pBft3oT1mEePm6Xdodg1tEaTRZfA34u6VFAwDrAnr0tEBETajSfXWfey4DL6kzrBjZvME4zM2uBhpJFRNwm6T3Au3PTrIh4tXVhmZlZJ2n0aqihwLeAr0bETGCUpF1bGpmZmXWMRk9wnwu8Amyfx+cBx7QkIjMz6ziNJouNIuIE4FWAfCWTazSZmQ0SjSaLVyS9FQgASRsBC1sWlZmZdZRGr4Y6EvgtMFLSBcAHgf1aFZSZmXWWRq+Guk7SNGA70uGnr0bEky2NzMzMOkajV0N9EPh7RPwaWAM4TNIGrQzMzMw6R6PnLE4DXpL0PlKhvwdI5cLNzGwQaDRZLMpF/3YDTo2IU4HVWheWmZl1kkZPcD8v6VDgX4EdcrnylVoXlpmZdZJGexZ7ki6V3T8iHiOVCv9+y6IyM7OO0mvPQtI1pEtmfxMRP6y0R8Tf8DkLM7NBo6xnsS/wDHCUpGmSTpO0m6RV+yE2MzPrEL32LPIhp8nA5HyeYltgZ+Cbkl4Grs1lQMzMbDnW6AluIuI14M/5dYSkYcAnWhWYmZl1jkZvyjtB0uqSVpJ0g6QeYHxEXNDi+MzMrAM0ejXUThGxANgVeBjYGPhGq4IyM7PO0miyqByu2gX4eUQ818hCks6R9ISkmYW2tSRdJ+n+/L5mbpekkyXNlnSnpK0Ky+yb579f0r4NxmxmZn2k0WRxtaR7ga2BGyR1AX9vYLnJwPiqtkOAGyJiNHBDHod04nx0fk0klRhB0lqkqrfbAtsAR1YSjJmZ9Y+GkkVEHAJ8ABibn739Eqn0R9lyNwNPVzXvBpyXh88Ddi+0nx/JVGANSeuSTqJfFxFPR8QzwHW8OQGZmVkLNfMM7i+Tf+0D6wFjl3Kba0fE/Dz8GLB2Hh4OzCnMNze31Ws3M7N+0uwzuD+Qx/vkGdy5OGEs63oqJE2U1C2pu6enp69Wa2Y26LXjGdyP58NL5Pcncvs8YGRhvhG5rV77m0TEGRExNiLGdnV1LWV4ZmZWrR3P4L6KVEaE/H5loX2ffFXUdsBz+XDVNcBOktbMJ7Z3ym1mZtZPWvoMbkkXAeOAYZLm5vUcB1wqaX/gEWCPPPsU4JPAbNIJ9C8ARMTTkr4L3JbnOzoiqk+am5lZC7X0GdwRMaHOpI/VmDeAg+qs5xzgnEZiNTOzvldWonyrqqbKVUzrS1o/Iqa1JiwzM+skZT2LE3uZFsBH+zAWMzPrUGUlynfsr0DMzKxzNXTOQtJbSDflfYjUo7gFOD0iGin5YWZmA1yjV0OdDzwP/DiPfw74KfDZVgRlZmadpdFksXlEbFoYv1HS3a0IyMzMOk+jN+VNyzfKASBpW6C7NSGZmVmnabRnsTXwJ0l/y+PrA7MkzSDdIrFFS6IzM7OO0GiycElwM7NBrNE7uB/JdZlGFpfxTXlmZoNDo5fOfpdUC+oB3igp7pvyzMwGiUYPQ+1BKlP+SiuDMTOzztTo1VAzgTVaGIeZmXWwRnsWxwJ3SJpJ4TkWEfGplkRlZmYdpdFkcR5wPDADeK114ZiZWSdqNFm8FBEntzQSMzPrWI0mi1skHUt69GnxMJQvnTUzGwQaTRZb5vftCm2+dNbMbJBo9KY8P9fCzGwQa7RngaRdgM2At1TaIuLoZjco6d3AJYWmdwFHkC7N/RLQk9sPi4gpeZlDgf2BxcBXIuKaZrdrZmZLr9E7uE8HhgI7AmcBnwFuXZoNRsQsYExe7xBgHnAF8AXgpIj4QdW2NwX2IiWq9YDrJW0SEYuXZvtmZta8Rm/K+0BE7AM8ExHfAbYHNumD7X8MeCAiHullnt2AiyNiYUQ8BMwGtumDbZuZWYMaTRYv5/eXJK0HLALW7YPt7wVcVBg/WNKdks7JhQsBhgNzCvPMzW1vImmipG5J3T09PbVmMTOzpdBosrha0hrACcDtwEMs+SXfNEkrA58Cfp6bTgM2Ih2img+c2Ow6I+KMiBgbEWO7urqWJTwzMyvo9ZyFpPcDcyLiu3n8baS7uO8FTlrGbe8MTIuIxwEq73k7ZwJX59F5pNLoFSNym5mZ9ZOynsVPgFcAJO0AHJfbngPOWMZtT6DQO5FUPKz1aVLxQkg3Au4laRVJGwKjWcqT62ZmtnTKroYaEhFP5+E9gTMi4jLgMknTl3ajklYF/hE4oNB8gqQxpJv9Hq5Mi4i7JF0K3E06V3KQr4QyM+tfpclC0ooRsYh05dLEJpatKyJeBN5R1fb5XuafBExa2u2ZmdmyKfvCvwi4SdKTpCuibgGQtDHpUJSZmQ0CvSaLiJgk6QbSZbLXRkTlkaorAP/R6uDMzKwzlB5KioipNdrua004ZmbWiRq9z8LMzAYxJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKtS1ZSHpY0gxJ0yV157a1JF0n6f78vmZul6STJc2WdKekrdoVt5nZYNTunsWOETEmIsbm8UOAGyJiNHBDHgfYGRidXxOB0/o9UjOzQazdyaLabsB5efg8YPdC+/mRTAXWkLRuG+IzMxuU2pksArhW0u2SJua2tSNifh5+DFg7Dw8H5hSWnZvbliBpoqRuSd09PT2titvMbNBZsY3b/lBEzJP0TuA6SfcWJ0ZESIpmVhgRZwBnAIwdO7apZc3MrL629SwiYl5+fwK4AtgGeLxyeCm/P5FnnweMLCw+IreZmVk/aEuykLSqpNUqw8BOwEzgKmDfPNu+wJV5+Cpgn3xV1HbAc4XDVWZm1mLtOgy1NnCFpEoMF0bEbyXdBlwqaX/gEWCPPP8U4JPAbOAl4Av9H7KZ2eDVlmQREQ8C76vR/hTwsRrtARzUD6GZmVkNnXbprJmZdSAnCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpfo9WUgaKelGSXdLukvSV3P7UZLmSZqeX58sLHOopNmSZkn6RH/HbGY22K3Yhm0uAr4eEdMkrQbcLum6PO2kiPhBcWZJmwJ7AZsB6wHXS9okIhb3a9RmZoNYv/csImJ+REzLw88D9wDDe1lkN+DiiFgYEQ8Bs4FtWh+pmZlVtPWchaRRwJbAX3LTwZLulHSOpDVz23BgTmGxudRJLpImSuqW1N3T09OqsM3MBp22JQtJbwMuA74WEQuA04CNgDHAfODEZtcZEWdExNiIGNvV1dWX4ZqZDWptSRaSViIligsi4nKAiHg8IhZHxGvAmbxxqGkeMLKw+IjcZmZm/aQdV0MJOBu4JyJ+WGhftzDbp4GZefgqYC9Jq0jaEBgN3Npf8ZqZWXuuhvog8HlghqTpue0wYIKkMUAADwMHAETEXZIuBe4mXUl1kK+EMjPrX/2eLCLiD4BqTJrSyzKTgEktC8rMzHrlO7jNzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1IDJllIGi9plqTZkg5pdzxmZoPJgEgWkoYApwI7A5sCEyRt2t6ozMwGjwGRLIBtgNkR8WBEvAJcDOzW5pjMzAaNFdsdQIOGA3MK43OBbatnkjQRmJhHX5A0qx9iW54MA55sdxDtoOPbHYH1sUH7WeY7WpalN6g3YaAki4ZExBnAGe2OY6CS1B0RY9sdh9my8me57w2Uw1DzgJGF8RG5zczM+sFASRa3AaMlbShpZWAv4Ko2x2RmNmgMiMNQEbFI0sHANcAQ4JyIuKvNYS2PfAjPlhf+LPcxRUS7YzAzsw43UA5DmZlZGzlZmJlZKSeLDiJpsaTpkmZK+rmk4Xl8uqTHJM0rjK+8DNtZW9LVkv4q6W5JU3L7KEkh6ZjCvMMkvSrplBrr2U9Sj6Q7JN0v6RpJHyhMnyzpobyd+ySdL2lEYfrDkmZIulPStZLWWdp9ss7QX5/hvK2dJXXnz/Adkk7sq/2os72jJX08D/9e0qC6NNfJorO8HBFjImJz4BVgzzw+BjgdOKkynu9kf5P8hf/7ku0cDVwXEe+LiE2BYq2th4BdCuOfBXq7mOCSiNgyIkYDxwGXS3pvYfo3IuJ9wLuBO4DfVX1J7BgRWwDdwGElcVvn65fPsKTNgVOAf82f4bHA7L7ckWoRcUREXL80y+aSRQOak0XnugXYuEXrXpd0FzwAEXFnYdpLwD2FX017Apc2stKIuJF0FcrEGtMiIk4CHiPV+Kp2M63bX2uPVn6GvwlMioh7ASJicUScBq8nm9/lHusNktbP7ZMlnSZpqqQHJY2TdI6keyRNrqxY0guSTpJ0V16+q7D8Z6oDkbSTpD9LmpZ7U2/L7Q9LOl7SNNKPrgHNyaIDSVqR9IU6o0WbOBU4W9KNkr4tab2q6RcDe0kaCSwGHm1i3dOA9yzF9F1p3f5aP+uHz/DmwO11pv0YOC/3WC8ATi5MWxPYHvhP0r1aJwGbAf8gaUyeZ1WgOyI2A24CjqwXhKRhwOHAxyNiK1IP+b8KszwVEVtFxMXN7V7nGRD3WQwib5U0PQ/fApzd6IKSrgA2BFYG1i+s538j4tzivBFxjaR3AeNJ/6HvyN36it8C3wUeBy5pch/KCtNUT79R0mLgTtJ/OhvY+uUzXGJ74J/z8E+BEwrTfhURIWkG8HhEzMjbvgsYBUwHXuONz/3PgMt72dZ2pErYf5REjv3PhenN/v/pWE4WneXlfGy3aRHxaUhdcGByRIwrmf9p4ELgQklXAzuQf6lFxCuSbge+TvqP8KkmQtkSuKdk+g2F8R0jYnAWfFs+9ddn+C5ga+CvTW5mYX5/rTBcGa/3fdjbzWginf+bUGf6i82F17l8GGoQkvRRSUPz8GrARsDfqmY7EfhWTiqNrvcjpPMVZ9aYJklfIZ0v+e3Sxm6WfR84TNImAJJWkHRgnvYnUkkggL1JPZxmrABUzk18DvhDL/NOBT4oaeMcx6qVmJY37lkMTlsDp0haRPqPcVZE3JZ/0QGQy6k0UlJlT0kfAoaSrqT6l4go9iy+L+l/8vSppJ5EzatgzBoVEXdK+hpwUf7hE8DVefJ/AOdK+gbQA3yhydW/CGwj6XDgCdJFHvXi6JG0X45jldx8OHBfk9vseC73YWZWIOmFiHhbu+PoND4MZWZmpdyzMDOzUu5ZmJlZKScLMzMr5WRhZmalnCys4+UKn7MK1UrfmdsPzFVrp0v6g6RNc/sHc12gbkmjc9saubJtzc+8pJUkHadUPXdarvVTq4ZVX+7Xn/L7KEkzW7mtBuN5vaqqWTXfZ2H9TtKqwCsR8WoTi+0dEd1VbRdGxOl5nZ8CfkgqYfJ14JOk8g0H5vHDge9FxGt11v9d0g2Dm0fEQklrAx9pIr6mRcQHyud6M0krRsSiFsRzRF+vsxGS1mrm5k9rD/csrB02Ae6T9IOqcuZNiYgFhdFVeaMsw6ukmwCHAq9K2ggYGRG/r7WefFPXl4D/iIiFed2PR8SlefqE3IOZKen4wnIvSPp+rk56vaRtci/owZy8Ks/8uDK33y/pyOLyNWIZktd5W+4dHZDbx0m6RdJVwN01lpmc45sh6T9z+5fyev4q6TJJQyW9XdIjlR5WvuN4Tu5ZvV5VVali6ndyL2uGpPfk9i5J1+V9Piuva1hez6/ztmZKqnsjW17P6pIOkHQr8N+9zWudwT2LQUbSScCONSZdHBHHSdqRVImz2kuVX8KSbgFWqzHPf0fE9fnO2b1rTL85Ir4SEXdI2oJ0Z+xZkoJUcO7SiKhXS+dcpYKDlwHHRL7mW9JBpCqfKwMfzfMeC5wPvAx8HvgBvRcp3Bj4W1XyIa9/PeB40l3vzwDXSto9In5JSlC/i4hvKBXBOwb4R1I9rfNIVU0BtiFVSX0JuE3Sr2v0kir2B56LiPfnO4L/KOnaPG0rUs/noaplxgDD8zMkkLRGbr88Is7MbccA+0fEj5UK9H0EuJFU7feaiHhVelMNyCcjYitJXyZ9oX+RVIH1dxFxrKTxOV5IPbpHI2KXvL2319o5pbv9vwh8kPRv+a8RcV+e1vLPnqR3U7+437iIeLbONIsIv/xq6wt4L6mez4I604fn99WAa4F9aszzOVJZ6ur2HUhfQJuQviR+BqxdNc8WwB11tr0bcH5hfH/gh3l4IW/cq3Q08O08vALwbB7er2r5o4Gv5eEX8vsoYGYe/gWpVMT0/HoI2AkYB9xYJ8Y1gQdIpbnHAyvk9o+Q6iLNyOs5vfC3qgxfAfxjHp4MfCYPP1z4u28LXJ+HpwMbFrb9NDAs/30fJiXWD9eJ82RSwv0cMKTdnzu/mnu5ZzHIdELPohDLKGBfYAKpeuhRtWKOiHn5/XlJF5J+qZ9fHT9wWrFB6afy4aSicj8mPTBnFPAV4NuFWWeTSmKvHjV6F714NfK3IIUqphHxmtLzHF7fhepd6mWdIh0Ou6ZqX8ZRp4JpRDwj6X3AJ0jnaPYA/o305b97RPxVqX7RuLzIVcD3JK1F6jH9rk4slaqsiyk5ChER90nainSu6BhJN0TE0VWz/RBYQOqdjJd0LvD7yt/QPYsO1+5s5dfge5G+sK8n/Ur9KvCOXuZdERiWh1ci/fI+MI+PLsz3T6QH1hSX3Zc3fsVfAYwEPkx6tGf1dk4AzgVWzuNdpKebrQs8Qvr1PCTHvVue54XC8keRvrAoTiP1LB4F1gLeSnpux9iqeUbxRs9iIvBLYKU8vgnpcNc44Oo6f6NhwOp5eHNgeh5+Enhn/rtdRyr7XVnm56RnPfxfoW0yS/YsKn/3saQvdUgPzvpWHt6JlPiGAesBb8ntuwK/7OXfdAgpqVwOzCJdvND2z6Vfvb/cs7B2WAwcFhG3NjDvKsA1klbijS/rSgn0g/Olnq+SDm/sW1kon7Tej/SFBulX7RTSc6E/V2M7h5POOdwt6e+kX/FHRMR8SYeQju8L+HVEXNnEvgLcSjo+PwL4WdQ/XwFwFil5TMs9ox5g95L1Dyed06lcsHJofv8f4C95HX9hyV/kl5ASxrhGdyL7DqnC6udJD/l5DHg+r+f7kl4j/Xv8e70VRMRi0r/FFKXLoJfLkt7LG9eGMmuhfPhnbEQc3O5Y+kI+6b44IhZJ2h44LZbyYUc2sLhnYWbNWB+4NPdiXiFdcmyDgHsWZmZWyjflmZlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZX6f0xAIcTawUzpAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "%matplotlib inline\n", "\n", "plt.title(\"Training Throughput \\n (Higher is better)\")\n", "plt.ylabel(\"Samples/sec\")\n", "\n", - "plt.bar(x=[1], height=native_throughput, label=\"Baseline PT\", width=0.35)\n", - "plt.bar(x=[1.5], height=optimized_throughput, label=\"Compiler-enhanced PT\", width=0.35)\n", + "plt.bar(x=[1], height=native_throughput, label=\"PT + SM DDP\", width=0.35)\n", + "plt.bar(x=[1.5], height=optimized_throughput, label=\"PT + Compiler\", width=0.35)\n", "\n", "plt.xlabel(\" ====> {} Compiler savings <====\".format(avg_speedup))\n", - "plt.xticks(ticks=[1, 1.5], labels=[\"Baseline PT\", \"Compiler-enhanced PT\"])" + "plt.xticks(ticks=[1, 1.5], labels=[\"PT + SM DDP\", \"PT + Compiler\"])" ] }, { "cell_type": "markdown", - "id": "5c21e6c8", + "id": "d1713454", "metadata": {}, "source": [ "### Convergence of Training Loss\n", @@ -557,10 +936,33 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "0734befd", + "execution_count": 29, + "id": "b02abff1", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5z0lEQVR4nO3deXxU9bn48c8zSSb7npCFEBJ2CEtYRBBxAQXrrtUqLlVrf7a2ttbrtbW3Xlt7W6/tba22dtFWq3Wl7orFHbQggiBbCCBbAgmBLITs68z398c5CQGTkG0yk5nn/Xqd18ycc+ac52TgOed8tyPGGJRSSgUOh7cDUEopNbg08SulVIDRxK+UUgFGE79SSgUYTfxKKRVgNPErpVSA0cSvBpWIrBSRbw7Svm4VkcMiUisiiR7cT62IjBrodZXyFE38asCJSIGINNhJ7rCIPCkiUb3cRpaIGBEJ7mMMIcCDwCJjTJQxpqLDsvl2bLUiUmfvp7bDlNmbfdnb3zvQ6/aGiPxMRJ4Z6O0q/6SJX3nKRcaYKGAGMAu4Z5D3nwKEAdtOXGCM+bedgKOAHHt2XNs8Y8z+tnX7euJRypdp4lceZYwpBpYDk09cJiIOEblHRApFpFRE/iEisfbij+3Xo/ZV+NxOvh8qIg+JyEF7esieNw7Y2eH7H/Y0XvvK+SUReUZEqoEbRWS2iKwRkaMiUiIij4iIs8N3jIiMsd8/KSJ/FJG3RKRGRNaKyOg+rrtIRHaKSJWI/ElEPupLMZmIXCwi2+z4V4rIxA7LfiQixfb+d4rIQnv+bBFZLyLV9l3bg73dr/JdmviVR4nICOB8YGMni2+0p7OBUUAU8Ii97Az7te1KfE0n3/8JMAfIBaYBs4F7jDFfcPyV/IJehn0J8BIQBzwLuIA7gCRgLrAQ+E43378auA+IB3YDv+ztuiKSZMfwYyAR60R2Wi+PA/sk+DzwAyAZ+Bfwpog4RWQ8cBtwijEmGlgMFNhffRh42BgTA4wG/tnbfSvfpYlfecprInIUWAV8BNzfyTrXAg8aY/YaY2qxktzVvSheuRb4uTGm1BhThpVAr+9/6KwxxrxmjHEbYxqMMRuMMZ8aY1qNMQXAo8CZ3Xz/VWPMOmNMK9aJI7cP654PbDPGvGIv+z1wqA/HchXwljHmPWNMC/AbIBzrJOICQoFJIhJijCkwxuyxv9cCjBGRJGNMrTHm0z7sW/koTfzKUy41xsQZY0YaY75jjGnoZJ10oLDD50IgGKt8vic6+356n6I93oGOH0RknIgsE5FDdvHP/VhX/13pmKDrse5kertuesc4jDWaYlEPYj/RcX8jY4zb3u5wY8xurDuBnwGlIvKCiLT9/W4GxgE7ROQzEbmwD/tWPkoTv/Kmg8DIDp8zgVbgMNCTYWM7+/7BAYjrxH3/GdgBjLWLPv4LkAHYT3dKgIy2DyIiHT/3wnF/I3s7I4BiAGPMc8aY0+11DPAre/4uY8wSYJg97yURiezboShfo4lfedPzwB0ikm0397wfWGoXbZQBbqyy/+6+f4+IJNtl4vcCnmjSGA1UA7UiMgG41QP7ONFbwBQRudQu+voukHqS7zhEJKzDFIpVNn+BiCy0m7jeCTQBn4jIeBFZYK/XCDRg/c0RketEJNm+Qzhqb9894EepvEITv/KmJ4CnsVrw7MNKPt8DMMbUY1V0rrZbo8zp5Pu/ANYDW4CtwOf2vIH2n8A1QA3wV2CpB/ZxHGNMOXAl8GugApiEdaxN3XxtCVbybpv2GGN2AtcBfwDKgYuwmto2Y5XvP2DPP4R1df9je1vnAdtEpBarovfqLorr1BAk+iAWpXyfiDiwyvivNcas8HY8amjTK36lfJSILBaROLsopq1eQVvXqH7TxK+U75oL7OFYEc2lWtyiBoIW9SilVIDRK36llAowQ2IAqqSkJJOVleXtMJRSakjZsGFDuTEm+cT5QyLxZ2VlsX79em+HoZRSQ4qIFHY2X4t6lFIqwGjiV0qpAKOJXymlAsyQKONXSg2+lpYWioqKaGxs9HYo6iTCwsLIyMggJCSkR+tr4ldKdaqoqIjo6GiysrKwBvVUvsgYQ0VFBUVFRWRnZ/foO1rUo5TqVGNjI4mJiZr0fZyIkJiY2Ks7M038SqkuadIfGnr7O/l34t/8Aqx/wttRKKWUT/HvxJ/3Mnz+D29HoZTqo6CgIHJzc5k8eTJXXnklxcXF5ObmkpubS2pqKsOHD2//3Nzc3Of9HD58mAsvvJBp06YxadIkzj//fAAKCgoQEe655572dcvLywkJCeG222770naefPJJkpOTmT59OmPHjmXx4sV88skn7ctvvPFGsrOzmTZtGuPGjePrX/86RUXHnqiZlZXFlClTmDp1KosWLeLQob48Zvnk/DvxO4LB3ertKJRSfRQeHs6mTZvIy8vD6XSydOlSNm3axKZNm/j2t7/NHXfc0f7Z6XR2uo2CggLOOuusbvdz7733cu6557J582by8/N54IEH2pdlZ2fz1ltvtX9+8cUXycnJ6XJbV111FRs3bmTXrl3cfffdXH755Wzfvr19+f/93/+xefNmdu7cyfTp01mwYMFxJ60VK1awZcsWZs2axf3333+yP1Gf+HfiFwe4Xd6OQik1AObPn8/u3bs9su2SkhIyMo490njq1Knt7yMiIpg4cWL7sDFLly7la1/7Wo+2e/bZZ3PLLbfw2GOPfWmZiHDHHXeQmprK8uXLv7T8jDPO8Njx+ndzTr3iV2pA3PfmNvIPVg/oNielx/DTi7q+cu6otbWV5cuXc9555w1oDG2++93vctVVV/HII49wzjnncNNNN5Gent6+/Oqrr+aFF14gJSWFoKAg0tPTOXjwYI+2PWPGDB599NFul+/YsYNLLrnkuPnLli1jypQpfTugkwiAxK9X/EoNVQ0NDeTm5gLWFf/NN9/c4+9edtll7Nu3j+bmZvbv39++ndtvv52bbrrpuHUXL17M3r17efvtt1m+fDnTp08nLy+vffl5553Hf//3f5OSksJVV13Vq2M42TNPTlx+9tlnExQUxNSpU/nFLzzxCOmASPx6xa9Uf/X0ynygtZXx98Wrr74KWGX8N954IytXrux2/YSEBK655hquueYaLrzwQj7++GNmzpwJgNPpZObMmfz2t78lPz+fN954o8dxbNy4kYkTJ3a7fOHChe2fV6xYQVJSUo+33xd+nviD9IpfKXVSH374IXPmzCEiIoKamhr27NlDZmbmcevceeednHnmmSQkJPR4ux999BGPPfYYK1as+NIyYwx/+MMfKCkp8VgRVlf8P/EbTfxKqe5t2LCB2267jeDgYNxuN9/85jc55ZRTKCgoaF8nJyen29Y8bZYuXcqqVauor68nOzubl19++bgr/rvuuov/+Z//ob6+njlz5rBixYouWyR5ypB45u6sWbNMnx7EsuwO2P4m3OWZmnGl/Nn27du7LaJQvqWz30tENhhjZp24rn8359QyfqWU+hL/TvyiZfxKKXUi/078WrmrlFJf4ueJX4t6lFLqRH6e+IM08Sul1An8OvHvKK23mnMOgZZLSik1WPw68W8/3GC9MW7vBqKU6pPBGpYZYPny5cyaNYtJkyYxffp07rzzzgE6is7de++9vP/++wCcddZZ9KnJeh95LPGLyBMiUioieR3mJYjIeyKyy36N99T+AYwEWW+0uEepIWmwhmXOy8vjtttu45lnniE/P5/169czZswYDxzRMT//+c8555xz+vRdl6t/jVY8ecX/JHBiP+S7gQ+MMWOBD+zPHiNBmviV8heeHJb517/+NT/5yU+YMGECYN1p3HrrrYB14liwYAFTp05l4cKF7N+/H7AeqnLrrbcyZ84cRo0axcqVK/nGN77BxIkTufHGG9u3HRUVxR133EFOTg4LFy6krKys/fsvvfTSl2J59913mTt3LjNmzODKK6+ktrYWsB7S8qMf/YgZM2bw4osv9ut4PTZkgzHmYxHJOmH2JcBZ9vungJXAjzwVgzjsw9MmnUr1z/K74dDWgd1m6hT4ygMnXw/PD8ucl5fXZdHO9773PW644QZuuOEGnnjiCb7//e/z2muvAVBZWcmaNWt44403uPjii1m9ejV/+9vfOOWUU9i0aRO5ubnU1dUxa9Ysfve73/Hzn/+c++67j0ceeaTTfZWXl/OLX/yC999/n8jISH71q1/x4IMPcu+99wKQmJjI559/3u/jHeyxelKMMSX2+0NASlcrisgtwC3AlwZL6rH2xK9X/EoNRYM1LHN31qxZwyuvvALA9ddfzw9/+MP2ZRdddBEiwpQpU0hJSWkfPz8nJ4eCggJyc3NxOBztQzlfd911XH755V3u69NPPyU/P5958+YB0NzczNy5c9uX93ZI6K54bZA2Y4wRkS6b2xhjHgMeA2usnr7sQxxtRT16xa9Uv/TwynygDdawzDk5OWzYsIFp06b1ah+hoaEAOByO9vdtn1tbO7/gFJEut2eM4dxzz+X555/vdHlkZGSv4uvKYLfqOSwiaQD2a6kndyZBdmWPq3+1/Uop/3bXXXdx//3388UXXwDgdrv5y1/+AsBpp53GCy+8AMCzzz7L/Pnze7Vtt9vdXpb/3HPPcfrpp3e57pw5c1i9enV7XUZdXV17TANpsBP/G8AN9vsbgNc9uTNXcLj1pqXek7tRSg1xU6dO5aGHHmLJkiVMnDiRyZMns3fvXgD+8Ic/8Pe//52pU6fy9NNP8/DDD/dq25GRkaxbt47Jkyfz4YcftpfXdyY5OZknn3ySJUuWMHXqVObOncuOHTv6dWyd8diwzCLyPFZFbhJwGPgp8BrwTyATKAS+Zow5crJt9XVY5peffZSv7voh3PIRpOf2+vtKBTIdlnlgREVFtbfM8aTeDMvsyVY9S7pYtLCL+QPOHRJhxdJcR9elakopFVj8uueusRO/u8nzZ1ullOrMYFzt91ZAJH5XU52XI1FqaBoKT+hTvf+d/Drx47SaPrkafe+Mq5SvCwsLo6KiQpO/jzPGUFFRQVhYWI+/49cPW28JS6LehBJUuApmf93b4Sg1pGRkZFBUVNQ+xIDyXWFhYWRkZPR4fb9O/EFhUWx0j2FW5T5vh6LUkBMSEkJ2dra3w1Ae4NdFPfERTiqJgvqTthhVSqmA4deJPyHSSaWJxtFQ4e1QlFLKZ/h54g/hCNEEN1WBSwdqU0op8PPEHx/hZLd7OIIbDm3xdjhKKeUT/Drxx0U42WxGWx8GeixxpZQaovw68Qc5hLqwNFrECRW7vB2OUkr5BL9O/ACxUeGUhgyHcs88sk0ppYYav0/8CRFOihzDoXzgx7RWSqmhyP8Tf6STPWY4VBZAS6O3w1FKKa8LiMSf1zocjEvL+ZVSigBI/PGRTjY2plofSrd7NxillPIBfp/4EyKc7HalYhwhUJrv7XCUUsrr/D/xRzppIZiW+NF6xa+UUgRI4geoixmrV/xKKUUAJP54O/EfiRwNR/eDPoZRKRXg/D7xJ0eHAlASmmXNKN/pvWCUUsoH+H/ij7IS/x7JtGaU7vBiNEop5X1+n/idwQ4SIp3sak6E4DAo0wpepVRg8/vEDzAsOpTDta2QNFZb9iilAl5gJP6YMEqrGyF5ohb1KKUCXkAk/pToUA5XN8GwCVBdBI3V3g5JKaW8JiAS/7CYUMpqm3AlTbBmlGnLHqVU4AqIxJ8SE4bLbTgaZT+NSyt4lVIBLCAS/7DoMABKJAWCw7WCVykV0AIj8cdYbflLa5shebwmfqVUQPNK4heRO0Rkm4jkicjzIhLmyf2lxFibL61ugmEToUxb9iilAtegJ34RGQ58H5hljJkMBAFXe3Kfbb13D1U3QvIEqCmBhqOe3KVSSvksbxX1BAPhIhIMRAAHPbkzZ7CDpCgnh6sbrSt+0Kt+pVTAGvTEb4wpBn4D7AdKgCpjzLsnricit4jIehFZX1ZW1u/9pseFU3zUvuIHLedXSgUsbxT1xAOXANlAOhApIteduJ4x5jFjzCxjzKzk5OR+7zc9NpyDRxsgdgQ4ozTxK6UCljeKes4B9hljyowxLcArwGme3ml6nJX4jYjVskfb8iulApQ3Ev9+YI6IRIiIAAsBj2fh4fHh1De7qGpogWGT4FAeGOPp3SqllM/xRhn/WuAl4HNgqx3DY57e7/A4q0ln8dEGyJwDDUe0glcpFZC80qrHGPNTY8wEY8xkY8z1xpgmT+8zPS4cgINHG2HkPGtmwSpP71YppXxOQPTchY6JvwHisyAmQxO/UiogBUziT4x04gx2WIlfBLLmQeFqLedXSgWcgEn8IsLwuHCrjB+s4p66Mij/wruBKaXUIAuYxA8wPC6cA5V24s863XrV4h6lVIDpVeIXEYeIxHgqGE/LSopgX1ktxhhIGAXRaVZxj1JKBZCTJn4ReU5EYkQkEsgD8kXkLs+HNvCyk6Kobmylsr7FKucfOQ8KtJxfKRVYenLFP8kYUw1cCizHGmrhek8G5SmjkiIB2Fdea83Imge1h6BijxejUkqpwdWTxB8iIiFYif8Ne5iFIXmJnG0n/r1ldfaMM63X3e97KSKllBp8PUn8jwIFQCTwsYiMBKo9GZSnZMSHE+wQ9pXbiT9xNAzLgfzXvBqXUkoNppMmfmPM740xw40x5xtLIXD2IMQ24IKDHGQmRhxL/AA5l8L+NVDt0UcCKKWUz+hJ5e7tduWuiMjjIvI5sGAQYvOIUUmRxyf+SZdar/lveCUepZQabD0p6vmGXbm7CIjHqth9wKNReVC2nfhdbruaInmcFvcopQJKTxK/2K/nA08bY7Z1mDfkjE+NoanVTUFFx+Key7S4RykVMHqS+DeIyLtYif8dEYkG3J4Ny3MmpkUDsL2kQ/10zqXWa97Lgx+QUkoNsp4k/puBu4FTjDH1gBO4yaNRedCYYVEEO+T4xJ80FjJPg7WPgavVe8EppdQg6EmrHjeQAdwjIr8BTjPGbPF4ZB4SGhzE6OQotpfUHL/gtNugaj9s10pepZR/60mrngeA24F8e/q+iNzv6cA8aVJ6DHnFVdaYPW3GnWeN37PmER3CQSnl13pS1HM+cK4x5gljzBPAecCFng3Ls6ZnxlFa03RsiGYARxDM+Q4Ub4ADa70XnFJKeVhPR+eM6/A+1gNxDKoZmfEAbCisPH5B7jUQHg8f/kKv+pVSfqsnif9/gY0i8qSIPAVsAH7p2bA8a0JqNBHOID4/MfE7I2HBf0PBv2Hri94JTimlPKwnlbvPA3OAV4CXgblYY/cMWcFBDqZnxrH+xMQPMPMmGD4T3vkvaOhkuVJKDXE9KuoxxpQYY96wp0PAkL8cnpkZz/aSamqbTmi+6XDAhb+D+gr44H+8E5xSSnlQXx+9OGR77raZMyoRt4E1eyq+vDBtGsz+Fqx/Qh/NqJTyO31N/EO+5nNWVgKRziBW7iztfIWz/wsSx8DS66G2i3WUUmoICu5qgYi8SecJXoBEj0U0SJzBDk4bk8TKnWUYYxA54SYmLAaufhb+Mh+W3QFXPWM9rlEppYa4LhM/8Js+Lhsyzpk4jPfyD5NXXM2UjE5aqSaPhwX3wHv/DR/9Cs66e/CDVEqpAdZl4jfGfDSYgXjD4pxU7nktjzc2F3ee+AHm3gYHN8LKB2DMOZAxa3CDVEqpAdbXMn6/EBfh5IyxySzbUoLb3UW1hcMBFz0M0Wnwyv/TJp5KqSEvoBM/wMW56ZRUNfLpvk5a97QJi4Ern4SjB+Clb+gInkqpIS3gE/+iSalEhwWz9LMD3a+YeSpc+CDs+RA+/PngBKeUUh7QXeUu0GXrnipgPfCoMaaxtzsVkTjgb8Bke9vfMMas6e12BkK4M4jLpw/n+c8O8LO6ZuIjnV2vPOPrVnn/6ochYzZMHNJj1SmlAlRPrvj3ArXAX+2pGqgBxtmf++Jh4G1jzARgGrC9j9sZEEtOzaS51c0rG4tPvvJ5D0D6dHj121C+2/PBKaXUAOtJ4j/NGHONMeZNe7oO62lc3wVm9HaHIhILnAE8DmCMaTbGHO3tdgbShNQYckfE8fy6/ceP0d+Z4FD42tMQFAL/vB6a67pfXymlfExPEn+UiGS2fbDfR9kfm/uwz2ygDPi7iGwUkb+JSOSJK4nILSKyXkTWl5WV9WE3vXPN7Ex2l9aybt+Rk68cNwK++lco3W517tIhnJVSQ0hPEv+dwCoRWSEiK4F/A/9pJ+un+rDPYKw7hT8bY6YDdVjP9D2OMeYxY8wsY8ys5OTkPuymdy6alk5MWDDPrN3fsy+MOQfO+jFsWWqN6aOUUkPESSt3jTH/EpGxwAR71s4OFboP9WGfRUCRMabtMVcv0UniH2zhziC+OjODZz4tpKxmEsnRoSf/0hl3QdFn8PbdVseutGmeD1Qppfqpp805ZwI5WBWxXxORr/d1h/awzgdEZLw9ayHWs3y97tpTR9LiMvxz/UmadrZxOODSP0NEIvzjEqudv1JK+biePGz9aayxeU4HTrGn/o5b8D3gWRHZAuQCPvHw9jHDopg7KpHn1u7H1VVP3hNFJcONb4GrxerZ29rk2SCVUqqfTlrUg5XkJ5mTNnfpOWPMJvp/8vCI6+aM5LvPfc5HX5SyYEJKz76UONoa1uHlm+H12+Dyx3QkT6WUz+pJUU8ekOrpQHzFopwUkqNDeebTHlbytplyBZx9D2z9pzWSp1JK+aieXPEnAfkisg5oL8cwxlzssai8KCTIwdWnjOCRFbs5cKSeEQkRPf/y/DuhfCes/F9InQITLvBcoEop1Uc9ueL/GXApVjn8bztMfmvJ7EwEeG5dL6/6HQ64+BFIy4WXbob9n3oiPKWU6peTJn5jzEedTYMRnLekx4WzaFIqz3xaSFVDS+++HBIG170M0SnwwjVQscczQSqlVB91mfhFZJX9WiMi1R2mGhGpHrwQveP7C8dS09jK46v29f7LkUnWoxrdrfD0pVDfg97ASik1SLpM/MaY0+3XaGNMTIcp2hgTM3ghesek9Bi+MjmVJ1bto7KuDyNTpE6B616BmkPw7BXQ6PfnSqXUENGjDlwiEiQi6SKS2TZ5OjBfcMe546hvbuWh97/o2wYyZsGVT0HJZnj2SmiqHdgAlVKqD3rSget7wGHgPeAte1rm4bh8wriUaK49dSTPrN3PzkM1fdvIhPPhq49D0Tp4/mpoaRjYIJVSqpd6csV/OzDeGJNjjJliT1M9HZiv+I9zxxEVGsx9b247+ZDNXcm5FC57FApWwQvXau9epZRX9STxH8B64lZAio908p+LxvHJngpe/rwHD2rpytSvwcV/gD0fwD9vgNa+jGitlFL915MOXHuBlSLyFsd34HrQY1H5mGtPHcmbm0u4781tnD4midTYsL5taMb14GqCt+6EV75pFQEFhQxssEopdRI9ueLfj1W+7wSiO0wBw+EQfn3FVFpcbv7r1a19L/IBOOWbsPh/If91eOar0HB0wOJUSqme6Ml4/PcNRiC+Lispkh8unsDPl+Wz9LMDXD27Hw2b5n4HwmLhze/DE4vh2hchLiAaSimlfEB3Hbgesl/fFJE3TpwGLUIfcuNpWZw+Jol7X9/GhsJ+dsqafq3Vzr+6BP66EIo3DEyQSil1EtJVsYWIzDTGbBCRMztbPpjDNsyaNcusX79+sHbXraP1zVzyx9XUNbl447Z5pMeF92+DZTutDl61ZdZzfCdeNDCBKqUCnohsMMZ8aQj87nrubrBfA26snu7ERTh5/IZZNLW4+H//WE99c2v/Npg8Hr75AaRMgqXXwfK7we0emGCVUqoTPenANVZEXhKRfBHZ2zYNRnC+asywaH6/ZDr5JdXc9eKW/lX2AkQNgxuWwYyvw9o/wwtLdHwfpZTH9KRVz9+BPwOtwNnAP4BnPBnUUHD2hGH8+CsTeGtrCf/3zs7+J39nBFz0e/jKr2H3B/DombBnxcAEq5RSHfQk8YcbYz7Aqg8oNMb8DNAnjAD/b/4orj5lBH9auYcHlu/of/IXgVO/Bd94GwRrZM+Vv4KBe+qlUkr1qANXk4g4gF0ichtQDER5NqyhQUS4/7IphAQ5ePTjvYQGO7jj3HFIf5+3mzELvr0a/nUXrLwfDnwKF/4O4rMGJG6lVGDr6Vg9EcD3gZnAdcANngxqKHE4hPsuzuFrszL4/Ye7+dHLW2huHYDK2bAYuOwvcMFv4cA6+NNc+OQRcPWzMlkpFfC6TfwiEgRcZYypNcYUGWNuMsZ81RijzxTswOEQHrh8Kt9bMIZ/ri/ihifWUVXfyyd3dUbE6un73bWQfQa8+xN4/Bw4tLX/21ZKBazuOnAFG2NcwOmDGM+Q5XAIdy4az4Nfm8aGwkou+9NqCsrrBmbjsRmw5AW44u9QVWRV/L73Ux3fXynVJ91d8a+zXzfavXWvF5HL26bBCG4ounxGBs9881Qq65u59E+rWbdvgJplisDky+G76yB3Cax+CB6aDBue0nb/Sqle6UkZfxhQASwALgQusl9VF2ZnJ/Dqd+aREOnk2r99yssbigZu4xEJcMkf4RvvQvJEe7yfRVD4ycDtQynl17pL/MNE5D+APGCr/brNfs0bhNiGtKykSF69dR6nZCVw54ub+cWyfBpbXAO3g8xT4aZ/WSeBqiL4+1fg6cu0/F8pdVLdJf4grGabUVjDMEedMKmTiI0I4alvzObrc0fyt1X7uPiRVeQVD+AzbURg+nXwnTVw9j2wfy385XT4+wVQuGbg9qOU8ivdDdL2uTFmxiDH0ylfGqStr1buLOVHL2+horaZ7y8cy3fOGk1wUI+edd9zDZWw6new6XmoK4Ux58Cos2D69RAeN7D7Ukr5vK4Gaesu8W80xkz3eGQ94A+JH6CqvoV738jj9U0HmZYRy2+unMbYFA8806a5DtY9Bh//FpprwBECZ90Ns2+x+gcopQJCXxJ/gjHGJ0YK85fE3+atLSXc89pW6ppd/Oeicdw0L5uQgb76B2iuh09+D1+8Awc/h/AEmHqV9SAYffCLUn6v14nf0+zOYeuBYmNMt62E/C3xA5TVNPHjV7by/vbDjBkWxX0X5zBvTJJndmaM9aCXlQ/Ang/BuCBuJMy73WoiGh7vmf0qpbzKFxP/fwCzgJhATPwAxhg+2F7Kz5fls/9IPedPSeUnF0xieH8f7tKdqiJYdgfsevfYvOQJcOaPYNKl4PDAnYdSyit8KvGLSAbwFPBL4D8CNfG3aWxx8deP9/LHlbsBuO3sMXxz/ijCQoI8t9PWJnj7btj1HriaofYwRCZbQ0TkXA7J4zy3b6XUoPC1xP8S8L9YzUT/s7PELyK3ALcAZGZmziwsLBzcIL2gqLKeX761neV5h8hMiODeCyexcOKw/o/2eTJuF2x/A9Y/Afs+tualTYN5P4DxX4EQD96BKKU8xmcSv4hcCJxvjPmOiJxFF4m/I3+/4j/Rv3eV8bM3trGnrI5TsxO4c9F4ZmcnDM7Oy76wWgTtWAY1JeCMhtQpMO1qmHIFOCMHJw6lVL/5UuL/X+B6rCd6hQExwCvGmOu6+k6gJX6A5lY3z60t5JEVeyivbWL+2CTuXDSe3BFxgxOA2wUFq2Dri9bU2gjB4TD2XJhwofVQeGfE4MSilOoTn0n8x+1cr/hPqqHZxdOfFvDnlXuorG/hnInDuOPcceSkxw5eEG43FK6C/Ndh+5tWfYAzyhoqOut0yL1GWwYp5YM08Q9xtU2tPLl6H499vJfqxlYumJLGD84Z65kOYN1xu2DncusksPNf0FwLIZEQOxwSx8DCe2HYxMGNSSnVKZ9M/D2lif+YqoYWHv/3Xh5ftY/6FheX5g7n9oVjyUryQtm7qxWKPoPNz8PnTx2bnzUfxiyE8edD0jhrTCGl1KDTxO9njtQ18+jHe3jqkwJaXIYrZmTwvYVjyIj3Urm7qwXKdsAXb8PmF6DCappKTAZMvBBGzIYx5+qQEUoNIk38fqq0ppE/rdjDc2v34zaGi6el860zRzM+dZCLgE5UscdqGpr/GhSsBrf9KMoRc2DC+TB6IaRO9mqISvk7Tfx+rqSqgb9+vI8XPttPfbOLBROG8a0zRjE7O8Hz/QBOpqkWPv0THN5m3RWU7bDmD5tk9RYetwhSp2mvYaUGmCb+AFFZ18zTnxby5CcFHKlrZnpmHN8+czTnTkzB4fCBsnZjoDQf9q60Wgjtt58bEJlsDSM99lwYvUBbCSk1ADTxB5iGZhcvbjjAYx/vpaiygVFJkdxyxigunT7cs0NB9FZVkVUUtPs92P2+9UwBcVjjB41bbD1PYOQ8CArxdqRKDTma+ANUq8vN8rxDPPrxHvKKq0mMdHLdnJFcP3ckSVGh3g7veG6XNYrorvesQeRKNlnzndHWnUD2fBh/AUSneDVMpYYKTfwBzhjDmj0VPL5qHx/sKMUZ7ODS3HRuPn2U9yuCu1JXAfs+su4Etr8JTdWAwPAZVuXwyLkwfCaEDWJnNqWGEE38qt2eslr+vnofL20oorHFzdxRiXxzfjZnjx/mG/UAnTHGqhTe9hrsXWH1HzBucARD9plW/UDmHEifrv0GlLJp4ldfUlnXzAufHeAfawooqWokKzGC6+aM5IqZGcRFOL0dXvfqymH/p3BgrXU3ULnPmh+RZBUJpc+w6gdScsDhQ3UaSg0iTfyqSy12PcDTawr4rKCS0GAHl8/I4MbTsny3GOhE1Qetp4vtWQGFq62RRcE6EYz/inU3MHoBxKQP7H4PrAMJgoyZA7tdpQaAJn7VI9tLqnnqkwJe3VhMU6ubU7LiuebUTL4yOc23WgN1x+2GonVW7+Ed/7IGmGusspbFZVongJzLIXMuBPfzzuZndv3Cz6r6tx2lPEATv+qVI3XNvLThAM+t3U9BRT1xESFcMSODJadmMjo5ytvh9Y7bZV2Zt/UkLt0OGGuY6cw5VtHQqLN7Xz9gDNwXZ73XxK98kCZ+1Sdut2HN3gqeW7ufd7YdotVtmDMqgWtPHcninFScwUOwt21jtdVcdN9HULQBSrdZ82MyrBZDbcNNJ0/o/kRQVw7/N9p6/6NCCI/zeOhK9YYmftVvpTWNvLi+iOfX7aeosoHESCdXzhrBNbMzyUwcwg9lqT5o9R3Y95F1Z1B1wJofmWw9dGb4TBh3HkQlH/+90h3wp1Ot97estO4YlPIhmvjVgHG7DR/vKuPZtfv5YPth3Abmj03i2lMzWTgxhZCgIXgX0MYYa0yhvSutFkO73oPWBmtZdBpMuMDqSZx9JpRthycvsJZNuwYu/gMEBXstdKVOpIlfeURJVQNLPzvA0s8OUFLVyLDoUK46ZQRXnTLCe0NEDyS3Gw5vhd0fWH0Hdn8AriZAAPv/ztjFsOsda9C5hT+FsYt0wDnlEzTxK49qdblZubOMZ9cWsvKLMgDOHj+Ma2ZncvaEYQT5asew3nK1QMlmq+nolqXQ0gC3fWbVGfzrh1BXahURjTrLqjAeffbANyFVqoc08atBc+BIvXUXsP4AZTVNpMWG8bVZI/jqjIyhXRdwIler1Xu4rUloa7P1OModb1lFRXWl1vyk8dZYQ2MWQsYpEDpE+kaoIU8Tvxp0LS437+cf5rl1+1m1uxxjYM6oBK6YOYLFOSlEh/nxiJvtdQUrrLuDglXgaraWJU+0OpWNmG3VFTj96GSofIomfuVVxUcbeG1jMf9cf4DCinrCQ4K4YGoaS2aPYEZmvPcfFuNpzXVQ+IlVYbxnBRz83LpbAKtuYNRZ1rOK06dDTJpXQ1X+QxO/8gnGGD7ff5SXNhzgjU0HqWt2MXZYFFedMoKLc9MZFh3m7RAHR1ONNepo2RfHBp1zt1rLolKsCuJRZ1l3BbEjdOA51Sea+JXPqWtqZdmWgzy/7gCbDhzFITBvTBKXTR/OopxUokIDqGlkQ6V1Etj1rlVEVLjaHoYaiBwGmadadwSZcyBlsg48p3pEE7/yabtLa3ht40Fe21RMUWUDYSEOFk1K5bLpwzl9bNLQ7hvQF24XFK2HgxutoSYOfAr1FdYyZ7TVwzhjllVZPHzWlzuXKYUmfjVEGGPYUFjJqxuLeWtrCUfrW0iMdHLh1DQunT6c3BFx/l8f0JWjB6xnFO9fY50UDm8D47KWxY2E9FzrriAtF5LG6hASShO/GnqaW9189EUZr20s5r3th2ludZOVGMElucO5dPpwspMivR2idzXXW30Kij6D4vXWyaC62FrmCIakcfadwWyrriBpvHYsCzCa+NWQVt3Ywtt5h3htYzFr9lZgDOSOiOPS3HTOn5oWOJXC3TEGSvOhZAsczoPyL6yTQkOltTw0FlKnQEIWDMuBtGkw4lQdZsKPaeJXfqOkqoE3Nx/k1Y0H2V5iVYDOzkrgkunpnJeTSqKvPUTem4yBij3W8wkOrLOGpC7feexk4AiBxNHWHUHGbEgYBWlTtZOZn9DEr/zSzkM1vJ13iNc3F7O3rI4ghzBnVAJfmZzGeZNTSdKTwJe5XVBbarUcOrTV6lxWtgOaa4+tEzfSelDNsAmQeZp1p6AdzYYcTfzKrxljyC+pZvnWQ/xrawl7y+vam4deMCWNBROHaXFQd1qboKrIujso2Wy1JireALWH7BXEenpZ8gTrZJA8wepspvUGPs1nEr+IjAD+AaRgDW/4mDHm4e6+o4lf9YYxhp2Ha1i2uYTXNxdz4EgDQQ5h7qhEFk9OZfGkFIbF6EmgR6qKrYrjsp1WMVHZTqjYdWz4CUcIJI6xHmqfPMEqNkoaa80LCfdu7MqnEn8akGaM+VxEooENwKXGmPyuvqOJX/WVMYa84mqW55Xwdt4h9pbXIQIzM+M5b3Iq505KYWRigLcO6i1Xq1VxfGAtVBZYJ4PDecceYAOAQPxI62SQNM56TR5vvQ+L8VbkAcdnEv+XAhB5HXjEGPNeV+to4lcDwRjDrtJa3s47xPK8Q+0Vw+NSojh3UgoLJqQwNSM28DqLDZTmejiyB8p3WSeGsh1Wb+SOdwgA0enWSSAuE2KGW3cJablWxbIWGw0on0z8IpIFfAxMNsZUd7WeJn7lCQeO1PNe/mHeyz/MuoIjuNyGpKhQ5o9NYnFOCvPGJPn3CKKDxdUKRwvtE8FOayq3X1vqj60XEmGdBMLiICEbEkZbnxNGWZXNoVFeO4ShyucSv4hEAR8BvzTGvNLJ8luAWwAyMzNnFhYWDnKEKpBU1Daxanc57+YfZtWucqoaWgh2CKeOSmBxTirzxyZrh7GBZgzUH7E6nZVstvoglH8BjdVwZC/Ulx+/fmSydQKIz7KKkeKzjn2OGa79ETrhU4lfREKAZcA7xpgHT7a+XvGrwdTicrOhsJKVO8t4O6+EggrrqnR8SjSnj03i9DFJzM5OIDKQBpHzhsYqq5VR5T6oLLTqE462vR44NlwFWD2VYzOOPxnEZUKQ07p7iBsZkHULPpP4xRpo5SngiDHmBz35jiZ+5S3GGPbbRUIf7ihlfWElza1uQoKEhRNSmD8uiXmjkxiZGBG4Ywh5g6vVulPoeDJoOzlUFnz5bgGsoqToNOsEEZlkvUalQni8dZKISICIRGvyk9FPfSnxnw78G9gK2E+i4L+MMf/q6jua+JWvaGxxsaGwkvfyD/N23iEOVTcCMDwunNNGJzJvTBKnjUnUPgPe1lRr9UtobbSKjY7utzqtVRdbU1Ux1Bzs/LsSBNGp1gkgapjVizku0z5JxFnzo1MhNMaeoiDYNzsK+kzi7wtN/MoXGWPYW17HJ7vLWb27gjV7K6hqaAFg7LAo5o1J4oxxSczOTgysZwsMFa1N1pPRGiqtu4TGo1BXYXVaqzlknSjqSq3lR/d3v62wWKtPQ2i0PcVYr85Iqz9D+xRxwusJ8wCCQo69rz9idZQL6duFhCZ+pTzM5TbkH6xm9Z5yVu8u57OCIzS2uAlyCJOHxzJ3VCKnZMUzZXisdiAbappqrSapjUehrtw6KTTVWA/LaXvvbrHWa6qxpypoabCneuu1tbH3+/7uZ5A8rk9ha+JXapC1FQut2VPBp3sr2Fx0lBaX9f9tWkYsp41JYnJ6LDNHxpMSE6p1BIHA7bKSf8eTQdtrs9201dV8rJlreByMmNPnpqxdJX69/1TKQ8JCgpg3Jol5Y5IAqG9uJa+4ms8KjvD+9sP89eO9tLqtE8HwuHBOzU5g+sh4po+IY3xqtHYk80eOIKv4x+ndpsF6xa+UlzS1uthRUsPn+yv5dG8FGworKa+1eriGBjuYnhnHKVkJTEqLYWZWvFYYq17Toh6lfJwxhqLKBjYXHeXzwqOsK6hge0kNLvuuIC02jKkZsUzNiGNaRhxTMmKJDdeexaprWtSjlI8TEUYkRDAiIYILp6YDVj1Bfkk1nxdWsqWoii1FR3ln2+H272QlRjA1I46pGbFMGxHHpLQY7VimTkr/hSjlw8JCgpiRGc+MzPj2eVX1LWwpPtp+Ivis4AhvbLbapItAdlIkOemx5KTHMDk9lknpMSREOr11CMoHaeJXaoiJjQhh/thk5o9Nbp9XWtPIlgNVbDtYTd7BKj4vrOTNzcc6KA2LDmVCWgyT0mKYlG69ZidFEuTQlkSBSBO/Un5gWHQY50wK45xJKe3zKuuayS+pZntJNdtLatheUs3je/a2NykNDXaQnRTJ+NRoctJjmJAaw7iUaG1aGgA08Svlp+Ijncc1JwVobnWzu7SWbQer+OJwDbtLa1m37wivbzp2dxAdFszo5CjGDItidHIUo5MjyUyMICM+Qnsg+wn9FZUKIM5gh1XUk378SJVH6prZeaiGXaU17Dpcy56yWv69q4yXNhQdt15SlJORiZGMTIhgZGIkWUkRZCZEkJUYSVxEiN4pDBGa+JVSJEQ6mTs6kbmjE4+bX9PYwt6yOg5U1nPgSAP7j9RRUF7Pp3sreHVTMR1bg0eHBZOVGMnIxAh7imREfAQZ8eGkxoZphzQfoolfKdWl6LAQpo2IY9qIuC8ta2xxUVRZT0F5PQUVdew/Uk9BRT1bi6tYnneovf8BWK2NUqLDSI8LIz0unOHx4WTEhZMRH0FaXBhpseHEhAXrHcMg0cSvlOqTsJAgxgyLZsyw6C8ta3G5Ka5soKiygeKj9RRXNnCwqpHiyga2Flfx7rbDNLvcx30nwhlEWqx1YkiLDWNYdBhpcWGkRIeREOUkOSqUlJgwnMF659BfmviVUgMuJMhBVlIkWV08rtLtNpTVNlFUWc/Bo40cqmrkYFUDJUcbKaluZOehMirqmo+7a4Bjdw5J0U6SokJJiLRek6KcJEaGkmi/JkQ5SYx0EhbiHw9UGWia+JVSg87hEFJiwkiJCWPmyM7XcbsNh2saKatpoqK2mcPVje13DWW11rxdh2spq22iudXd6TYinEGkxoYRGx5CXHgIcRFOYsNDiI9wEhcRYk9O4sJDiA0PITosmOiwEL+/q9DEr5TySQ6HkBYbTlpseLfrGWOobWqloraZijrrhHCkrpmKOuu1uLKBuuZWymub2VVaS1V9CzVNrd1uMzTYQXRY24nAmqJCg9vnRYUGExYSRHhIEOHOY68RziDCQoIIDXYQ7HDgDBbCncGEBTsIDQki2CGEBDm83nFOE79SakgTETshh3RZtHSiFpebqoYWjtY3c7S+hcr6FqobWqhpbKG6sZXaplZqGluoaWy1pxbKapraP9c1t9Kf8S1FINghBDscBAeJ9T7IQYhDCAoSQhwORKCh2cXSb81lREJE33fWCU38SqmAExLksOsG+vasXGMMTa1uGltcNLS4aGh2Ud9svW9scdHc6qbFZWh2uWlobm1ft8VlaHUZXG43LW5Dq8tNq9ua1+p226+GFpcbA4QFBxEcNPB3B5r4lVKql0SEsBCrWCfO28H0gX/XYCillPoSTfxKKRVgNPErpVSA0cSvlFIBRhO/UkoFGE38SikVYDTxK6VUgNHEr5RSAUZMf/odDxIRKQMK+/j1JKB8AMMZCvSYA0OgHXOgHS/0/5hHGmOST5w5JBJ/f4jIemPMLG/HMZj0mANDoB1zoB0veO6YtahHKaUCjCZ+pZQKMIGQ+B/zdgBeoMccGALtmAPteMFDx+z3ZfxKKaWOFwhX/EoppTrQxK+UUgHGbxO/iJwnIjtFZLeI3O3teAaKiIwQkRUiki8i20Tkdnt+goi8JyK77Nd4e76IyO/tv8MWEZnh3SPoOxEJEpGNIrLM/pwtImvtY1sqIk57fqj9ebe9PMurgfeRiMSJyEsiskNEtovIXH//nUXkDvvfdZ6IPC8iYf72O4vIEyJSKiJ5Heb1+ncVkRvs9XeJyA29icEvE7+IBAF/BL4CTAKWiMgk70Y1YFqBO40xk4A5wHftY7sb+MAYMxb4wP4M1t9grD3dAvx58EMeMLcD2zt8/hXwO2PMGKASuNmefzNQac//nb3eUPQw8LYxZgIwDevY/fZ3FpHhwPeBWcaYyUAQcDX+9zs/CZx3wrxe/a4ikgD8FDgVmA38tO1k0SPGGL+bgLnAOx0+/xj4sbfj8tCxvg6cC+wE0ux5acBO+/2jwJIO67evN5QmIMP+D7EAWAYIVo/G4BN/c+AdYK79PtheT7x9DL083lhg34lx+/PvDAwHDgAJ9u+2DFjsj78zkAXk9fV3BZYAj3aYf9x6J5v88oqfY/+A2hTZ8/yKfWs7HVgLpBhjSuxFh4AU+72//C0eAn4IuO3PicBRY0yr/bnjcbUfs728yl5/KMkGyoC/28VbfxORSPz4dzbGFAO/AfYDJVi/2wb8+3du09vftV+/t78mfr8nIlHAy8APjDHVHZcZ6xLAb9rpisiFQKkxZoO3YxlEwcAM4M/GmOlAHcdu/wG//J3jgUuwTnrpQCRfLhLxe4Pxu/pr4i8GRnT4nGHP8wsiEoKV9J81xrxizz4sImn28jSg1J7vD3+LecDFIlIAvIBV3PMwECciwfY6HY+r/Zjt5bFAxWAGPACKgCJjzFr780tYJwJ//p3PAfYZY8qMMS3AK1i/vT//zm16+7v26/f218T/GTDWbg3gxKogesPLMQ0IERHgcWC7MebBDoveANpq9m/AKvtvm/91u3XAHKCqwy3lkGCM+bExJsMYk4X1W35ojLkWWAFcYa924jG3/S2usNcfUlfGxphDwAERGW/PWgjk48e/M1YRzxwRibD/nbcds9/+zh309nd9B1gkIvH2ndIie17PeLuSw4OVJ+cDXwB7gJ94O54BPK7TsW4DtwCb7Ol8rLLND4BdwPtAgr2+YLVw2gNsxWox4fXj6MfxnwUss9+PAtYBu4EXgVB7fpj9ebe9fJS34+7jseYC6+3f+jUg3t9/Z+A+YAeQBzwNhPrb7ww8j1WH0YJ1Z3dzX35X4Bv2se8GbupNDDpkg1JKBRh/LepRSinVBU38SikVYDTxK6VUgNHEr5RSAUYTv1JKBRhN/EoBIuISkU0dpgEb0VVEsjqOxKiUtwWffBWlAkKDMSbX20EoNRj0il+pbohIgYj8WkS2isg6ERljz88SkQ/tMdI/EJFMe36KiLwqIpvt6TR7U0Ei8ld7rPl3RSTcawelAp4mfqUs4ScU9VzVYVmVMWYK8AjWKKEAfwCeMsZMBZ4Ffm/P/z3wkTFmGtbYOtvs+WOBPxpjcoCjwFc9ejRKdUN77ioFiEitMSaqk/kFwAJjzF57cLxDxphEESnHGj+9xZ5fYoxJEpEyIMMY09RhG1nAe8Z6yAYi8iMgxBjzi0E4NKW+RK/4lTo508X73mjq8N6F1q8pL9LEr9TJXdXhdY39/hOskUIBrgX+bb//ALgV2p8RHDtYQSrVU3rVoZQlXEQ2dfj8tjGmrUlnvIhswbpqX2LP+x7W07HuwnpS1k32/NuBx0TkZqwr+1uxRmJUymdoGb9S3bDL+GcZY8q9HYtSA0WLepRSKsDoFb9SSgUYveJXSqkAo4lfKaUCjCZ+pZQKMJr4lVIqwGjiV0qpAPP/AaHmlzwFHdM8AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "vanilla_loss = [i[\"loss\"] for i in n[\"train\"]]\n", "vanilla_epochs = [i[\"epoch\"] for i in n[\"train\"]]\n", @@ -570,14 +972,14 @@ "plt.title(\"Plot of Training Loss\")\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Training Loss\")\n", - "plt.plot(vanilla_epochs, vanilla_loss, label=\"Baseline PT\")\n", - "plt.plot(optimized_epochs, optimized_loss, label=\"Compiler-enhanced PT\")\n", + "plt.plot(vanilla_epochs, vanilla_loss, label=\"PT + SM DDP\")\n", + "plt.plot(optimized_epochs, optimized_loss, label=\"PT + Compiler\")\n", "plt.legend()" ] }, { "cell_type": "markdown", - "id": "323fb548", + "id": "1842f1f9", "metadata": {}, "source": [ "### Training Stats\n", @@ -588,22 +990,94 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7f757dc0", + "execution_count": 30, + "id": "9a196443", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
train_runtimetrain_samples_per_secondtrain_steps_per_secondtrain_lossepoch
PT + SM DDP1305.67741216.2272.2982.3402011000.0
PT + Compiler852.87781861.9321.1733.2869791000.0
\n", + "
" + ], + "text/plain": [ + " train_runtime train_samples_per_second \\\n", + "PT + SM DDP 1305.6774 1216.227 \n", + "PT + Compiler 852.8778 1861.932 \n", + "\n", + " train_steps_per_second train_loss epoch \n", + "PT + SM DDP 2.298 2.340201 1000.0 \n", + "PT + Compiler 1.173 3.286979 1000.0 " + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "import pandas as pd\n", "\n", - "pd.DataFrame([n[\"summary\"], o[\"summary\"]], index=[\"Native\", \"Compiler-enhanced\"])" + "pd.DataFrame([n[\"summary\"], o[\"summary\"]], index=[\"PT + SM DDP\", \"PT + Compiler\"])" ] }, { "cell_type": "code", - "execution_count": null, - "id": "f45b0aab", + "execution_count": 31, + "id": "34982525", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SageMaker Training Compiler is about 34% faster in terms of total training time.\n" + ] + } + ], "source": [ "speedup = (\n", " (n[\"summary\"][\"train_runtime\"] - o[\"summary\"][\"train_runtime\"])\n", @@ -617,7 +1091,7 @@ }, { "cell_type": "markdown", - "id": "266f2257", + "id": "8ec367e0", "metadata": {}, "source": [ "### Total Billable Time\n", @@ -627,8 +1101,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "2a7b17be", + "execution_count": 32, + "id": "fa66ac61", "metadata": {}, "outputs": [], "source": [ @@ -642,31 +1116,84 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "e3d3967f", + "execution_count": 33, + "id": "66b1c0b1", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
PT + SM DDPPT + Compiler
BillableSecs25821980
\n", + "
" + ], + "text/plain": [ + " PT + SM DDP PT + Compiler\n", + "BillableSecs 2582 1980" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "Billable = {}\n", - "Billable[\"Native\"] = BillableTimeInSeconds(native_estimator.latest_training_job.name)\n", - "Billable[\"Optimized\"] = BillableTimeInSeconds(optimized_estimator.latest_training_job.name)\n", + "Billable[\"PT + SM DDP\"] = BillableTimeInSeconds(native_estimator.latest_training_job.name)\n", + "Billable[\"PT + Compiler\"] = BillableTimeInSeconds(optimized_estimator.latest_training_job.name)\n", "pd.DataFrame(Billable, index=[\"BillableSecs\"])" ] }, { "cell_type": "code", - "execution_count": null, - "id": "d5deb4d6", + "execution_count": 34, + "id": "7e0373db", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SageMaker Training Compiler integrated PyTorch was 23% faster in summary.\n" + ] + } + ], "source": [ - "speedup = (Billable[\"Native\"] - Billable[\"Optimized\"]) * 100 / Billable[\"Native\"]\n", + "speedup = (Billable[\"PT + SM DDP\"] - Billable[\"PT + Compiler\"]) * 100 / Billable[\"PT + SM DDP\"]\n", "print(f\"SageMaker Training Compiler integrated PyTorch was {int(speedup)}% faster in summary.\")" ] }, { "cell_type": "markdown", - "id": "0527c59e", + "id": "ef17c2ac", "metadata": {}, "source": [ "## Clean up\n", @@ -676,8 +1203,8 @@ }, { "cell_type": "code", - "execution_count": null, - "id": "7aa8a7dc", + "execution_count": 35, + "id": "b96c6215", "metadata": {}, "outputs": [], "source": [ @@ -698,7 +1225,7 @@ }, { "cell_type": "markdown", - "id": "6d48d974", + "id": "26636a53", "metadata": {}, "source": [ "Also, to find instructions on cleaning up resources, see [Clean Up](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html) in the *Amazon SageMaker Developer Guide*." From eb668a7ef12283b034d0e3ea79fa3af5f19612a4 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Thu, 13 Oct 2022 23:23:12 +0000 Subject: [PATCH 5/9] turn off amp --- ...nguage-modeling-multi-gpu-multi-node.ipynb | 682 +++--------------- 1 file changed, 88 insertions(+), 594 deletions(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index 68b71253d2..63a9eec288 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "98f52af1", + "id": "930e721f", "metadata": {}, "source": [ "# Compile and Train the GPT2 Model using the Transformers Trainer API with the SST2 Dataset for Multi-Node Multi-GPU Training" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "84e26b8a", + "id": "56173568", "metadata": {}, "source": [ "1. [Introduction](#Introduction) \n", @@ -25,7 +25,7 @@ }, { "cell_type": "markdown", - "id": "392d66aa", + "id": "a7a7be64", "metadata": {}, "source": [ "## SageMaker Training Compiler Overview\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "be898d9d", + "id": "41df9c79", "metadata": {}, "source": [ "## Development Environment " @@ -55,7 +55,7 @@ }, { "cell_type": "markdown", - "id": "6346bd12", + "id": "6e1e5d50", "metadata": {}, "source": [ "### Installation" @@ -63,7 +63,7 @@ }, { "cell_type": "markdown", - "id": "d49ec5c5", + "id": "66ee6d74", "metadata": {}, "source": [ "This example notebook requires the **SageMaker Python SDK v2.108.0** and **transformers v4.21**." @@ -71,331 +71,30 @@ }, { "cell_type": "code", - "execution_count": 1, - "id": "4e4b43d1", + "execution_count": null, + "id": "6e4966b4", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n", - "Requirement already satisfied: sagemaker>=2.108.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (2.112.2)\n", - "Requirement already satisfied: botocore in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.27.89)\n", - "Collecting botocore\n", - " Downloading botocore-1.27.90-py3-none-any.whl (9.2 MB)\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.2/9.2 MB\u001b[0m \u001b[31m49.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", - "\u001b[?25hRequirement already satisfied: boto3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.24.89)\n", - "Collecting boto3\n", - " Downloading boto3-1.24.90-py3-none-any.whl (132 kB)\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.5/132.5 KB\u001b[0m \u001b[31m32.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", - "\u001b[?25hRequirement already satisfied: awscli in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.25.90)\n", - "Collecting awscli\n", - " Downloading awscli-1.25.91-py3-none-any.whl (3.9 MB)\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.9/3.9 MB\u001b[0m \u001b[31m83.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n", - "\u001b[?25hRequirement already satisfied: s3fs in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (0.4.2)\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.8.2-py3-none-any.whl (27 kB)\n", - "Requirement already satisfied: typing-extensions in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (4.4.0)\n", - "Requirement already satisfied: torch==1.11.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.11.0)\n", - "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.5.0)\n", - "Requirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (1.23.4)\n", - "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.1.5)\n", - "Requirement already satisfied: attrs<23,>=20.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (21.2.0)\n", - "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (21.3)\n", - "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (1.0.1)\n", - "Requirement already satisfied: schema in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.7.5)\n", - "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (4.8.2)\n", - "Requirement already satisfied: protobuf<4.0,>=3.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (3.20.1)\n", - "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.2.0)\n", - "Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sagemaker>=2.108.0) (0.2.8)\n", - "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (1.26.8)\n", - "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (0.10.0)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from botocore) (2.8.2)\n", - "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from boto3) (0.6.0)\n", - "Requirement already satisfied: PyYAML<5.5,>=3.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (5.4.1)\n", - "Requirement already satisfied: rsa<4.8,>=3.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (4.7.2)\n", - "Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (0.4.3)\n", - "Requirement already satisfied: docutils<0.17,>=0.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from awscli) (0.15.2)\n", - "Collecting aiobotocore~=2.4.0\n", - " Using cached aiobotocore-2.4.0-py3-none-any.whl (65 kB)\n", - "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (3.8.1)\n", - "Collecting fsspec==2022.8.2\n", - " Using cached fsspec-2022.8.2-py3-none-any.whl (140 kB)\n", - "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas) (2021.3)\n", - "Requirement already satisfied: aioitertools>=0.5.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiobotocore~=2.4.0->s3fs) (0.8.0)\n", - "INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.\n", - "Collecting numpy\n", - " Using cached numpy-1.23.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)\n", - "INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.\n", - "Collecting pandas\n", - " Using cached pandas-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)\n", - "INFO: pip is looking at multiple versions of typing-extensions to determine which version is compatible with other requirements. This could take a while.\n", - "Collecting typing-extensions\n", - " Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)\n", - "INFO: pip is looking at multiple versions of fsspec to determine which version is compatible with other requirements. This could take a while.\n", - "INFO: pip is looking at multiple versions of s3fs to determine which version is compatible with other requirements. This could take a while.\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.8.1-py3-none-any.whl (27 kB)\n", - "Collecting fsspec==2022.8.1\n", - " Using cached fsspec-2022.8.1-py3-none-any.whl (140 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.8.0-py3-none-any.whl (27 kB)\n", - "Collecting fsspec==2022.8.0\n", - " Using cached fsspec-2022.8.0-py3-none-any.whl (140 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.7.1-py3-none-any.whl (27 kB)\n", - "Collecting fsspec==2022.7.1\n", - " Using cached fsspec-2022.7.1-py3-none-any.whl (141 kB)\n", - "Collecting aiobotocore~=2.3.4\n", - " Using cached aiobotocore-2.3.4-py3-none-any.whl (64 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.7.0-py3-none-any.whl (27 kB)\n", - "Collecting fsspec==2022.7.0\n", - " Using cached fsspec-2022.7.0-py3-none-any.whl (141 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.5.0-py3-none-any.whl (27 kB)\n", - "Collecting fsspec==2022.5.0\n", - " Using cached fsspec-2022.5.0-py3-none-any.whl (140 kB)\n", - "Requirement already satisfied: aiobotocore~=2.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (2.3.3)\n", - "Collecting aiobotocore~=2.3.0\n", - " Using cached aiobotocore-2.3.2.tar.gz (104 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-2.3.1.tar.gz (65 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-2.3.0.tar.gz (65 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting s3fs\n", - " Using cached s3fs-2022.3.0-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2022.3.0\n", - " Using cached fsspec-2022.3.0-py3-none-any.whl (136 kB)\n", - "Collecting aiobotocore~=2.2.0\n", - " Using cached aiobotocore-2.2.0.tar.gz (59 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting s3fs\n", - " Using cached s3fs-2022.2.0-py3-none-any.whl (26 kB)\n", - "Collecting aiobotocore~=2.1.0\n", - " Using cached aiobotocore-2.1.2.tar.gz (58 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting fsspec==2022.02.0\n", - " Using cached fsspec-2022.2.0-py3-none-any.whl (134 kB)\n", - "Collecting aiobotocore~=2.1.0\n", - " Using cached aiobotocore-2.1.1.tar.gz (57 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-2.1.0.tar.gz (54 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hINFO: pip is looking at multiple versions of fsspec to determine which version is compatible with other requirements. This could take a while.\n", - "INFO: pip is looking at multiple versions of s3fs to determine which version is compatible with other requirements. This could take a while.\n", - "Collecting s3fs\n", - " Using cached s3fs-2022.1.0-py3-none-any.whl (25 kB)\n", - "Collecting fsspec==2022.01.0\n", - " Using cached fsspec-2022.1.0-py3-none-any.whl (133 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.11.1-py3-none-any.whl (25 kB)\n", - "Requirement already satisfied: fsspec==2021.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from s3fs) (2021.11.1)\n", - "Collecting aiobotocore~=2.0.1\n", - " Using cached aiobotocore-2.0.1-py3-none-any.whl\n", - "Requirement already satisfied: wrapt>=1.10.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiobotocore~=2.0.1->s3fs) (1.13.3)\n", - "Collecting fsspec==2021.11.1\n", - " Using cached fsspec-2021.11.1-py3-none-any.whl (132 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.11.0-py3-none-any.whl (25 kB)\n", - "Collecting aiobotocore~=1.4.1\n", - " Using cached aiobotocore-1.4.2.tar.gz (52 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting fsspec==2021.11.0\n", - " Using cached fsspec-2021.11.0-py3-none-any.whl (132 kB)\n", - "Collecting aiobotocore~=1.4.1\n", - " Using cached aiobotocore-1.4.1.tar.gz (52 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting s3fs\n", - " Using cached s3fs-2021.10.1-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2021.10.1\n", - " Using cached fsspec-2021.10.1-py3-none-any.whl (125 kB)\n", - "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.10.0-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2021.10.0\n", - " Using cached fsspec-2021.10.0-py3-none-any.whl (125 kB)\n", - "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.9.0-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2021.09.0\n", - " Using cached fsspec-2021.9.0-py3-none-any.whl (123 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.8.1-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2021.08.1\n", - " Using cached fsspec-2021.8.1-py3-none-any.whl (119 kB)\n", - "Collecting aiobotocore~=1.4.0\n", - " Using cached aiobotocore-1.4.0.tar.gz (51 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25hCollecting s3fs\n", - " Using cached s3fs-2021.8.0-py3-none-any.whl (26 kB)\n", - "Collecting fsspec==2021.07.0\n", - " Using cached fsspec-2021.7.0-py3-none-any.whl (118 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.7.0-py3-none-any.whl (25 kB)\n", - "Collecting aiobotocore>=1.0.1\n", - " Using cached aiobotocore-2.0.0.tar.gz (52 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.3.3.tar.gz (50 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.3.2.tar.gz (49 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.3.1.tar.gz (48 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.3.0.tar.gz (48 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.2.2.tar.gz (48 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.2.1.tar.gz (48 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.2.0.tar.gz (47 kB)\n", - " Preparing metadata (setup.py) ... \u001b[?25ldone\n", - "\u001b[?25h Using cached aiobotocore-1.1.2-py3-none-any.whl (45 kB)\n", - " Using cached aiobotocore-1.1.1-py3-none-any.whl (45 kB)\n", - " Using cached aiobotocore-1.1.0-py3-none-any.whl (43 kB)\n", - " Using cached aiobotocore-1.0.7-py3-none-any.whl (42 kB)\n", - " Using cached aiobotocore-1.0.6-py3-none-any.whl (42 kB)\n", - " Using cached aiobotocore-1.0.5-py3-none-any.whl (42 kB)\n", - " Using cached aiobotocore-1.0.4-py3-none-any.whl (41 kB)\n", - " Using cached aiobotocore-1.0.3-py3-none-any.whl (40 kB)\n", - " Using cached aiobotocore-1.0.2-py3-none-any.whl (40 kB)\n", - " Using cached aiobotocore-1.0.1-py3-none-any.whl (40 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.6.1-py3-none-any.whl (25 kB)\n", - "Collecting fsspec==2021.06.1\n", - " Using cached fsspec-2021.6.1-py3-none-any.whl (115 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.6.0-py3-none-any.whl (24 kB)\n", - "Collecting fsspec==2021.06.0\n", - " Using cached fsspec-2021.6.0-py3-none-any.whl (114 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.5.0-py3-none-any.whl (24 kB)\n", - "Collecting fsspec==2021.05.0\n", - " Using cached fsspec-2021.5.0-py3-none-any.whl (111 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-2021.4.0-py3-none-any.whl (23 kB)\n", - "Collecting fsspec==2021.04.0\n", - " Using cached fsspec-2021.4.0-py3-none-any.whl (108 kB)\n", - "Collecting s3fs\n", - " Using cached s3fs-0.6.0-py3-none-any.whl (23 kB)\n", - " Using cached s3fs-0.5.2-py3-none-any.whl (22 kB)\n", - " Using cached s3fs-0.5.1-py3-none-any.whl (21 kB)\n", - " Using cached s3fs-0.5.0-py3-none-any.whl (21 kB)\n", - "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker>=2.108.0) (3.6.0)\n", - "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->sagemaker>=2.108.0) (3.0.6)\n", - "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker>=2.108.0) (1.16.0)\n", - "Requirement already satisfied: pyasn1>=0.1.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from rsa<4.8,>=3.1.2->awscli) (0.4.8)\n", - "Requirement already satisfied: pox>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.3.0)\n", - "Requirement already satisfied: multiprocess>=0.70.12 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.70.12.2)\n", - "Requirement already satisfied: dill>=0.3.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (0.3.4)\n", - "Requirement already satisfied: ppft>=1.6.6.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathos->sagemaker>=2.108.0) (1.6.6.4)\n", - "Requirement already satisfied: contextlib2>=0.5.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from schema->sagemaker>=2.108.0) (21.6.0)\n", - "Installing collected packages: botocore, boto3, awscli\n", - " Attempting uninstall: botocore\n", - " Found existing installation: botocore 1.27.89\n", - " Uninstalling botocore-1.27.89:\n", - " Successfully uninstalled botocore-1.27.89\n", - " Attempting uninstall: boto3\n", - " Found existing installation: boto3 1.24.89\n", - " Uninstalling boto3-1.24.89:\n", - " Successfully uninstalled boto3-1.24.89\n", - " Attempting uninstall: awscli\n", - " Found existing installation: awscli 1.25.90\n", - " Uninstalling awscli-1.25.90:\n", - " Successfully uninstalled awscli-1.25.90\n", - "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", - "aiobotocore 2.3.3 requires botocore<1.24.22,>=1.24.21, but you have botocore 1.27.90 which is incompatible.\u001b[0m\u001b[31m\n", - "\u001b[0mSuccessfully installed awscli-1.25.91 boto3-1.24.90 botocore-1.27.90\n", - "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.\n", - "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n", - "\u001b[0m" - ] - } - ], + "outputs": [], "source": [ "!pip install \"sagemaker>=2.108.0\" botocore boto3 awscli s3fs typing-extensions \"torch==1.11.0\" pandas numpy --upgrade" ] }, { "cell_type": "code", - "execution_count": 2, - "id": "0c3adc03", + "execution_count": null, + "id": "3469acdb", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n", - "Requirement already satisfied: transformers==4.21 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (4.21.0)\n", - "Requirement already satisfied: datasets in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (2.6.0)\n", - "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (2021.11.10)\n", - "Requirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (5.4.1)\n", - "Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (2.26.0)\n", - "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (21.3)\n", - "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (3.4.0)\n", - "Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (0.12.1)\n", - "Requirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (4.62.3)\n", - "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (0.9.0)\n", - "Requirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from transformers==4.21) (1.23.4)\n", - "Requirement already satisfied: dill<0.3.6 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.3.4)\n", - "Requirement already satisfied: xxhash in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (3.0.0)\n", - "Requirement already satisfied: aiohttp in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (3.8.1)\n", - "Requirement already satisfied: fsspec[http]>=2021.11.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (2021.11.1)\n", - "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (1.5.0)\n", - "Requirement already satisfied: pyarrow>=6.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (7.0.0)\n", - "Requirement already satisfied: multiprocess in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.70.12.2)\n", - "Requirement already satisfied: responses<0.19 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets) (0.18.0)\n", - "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers==4.21) (4.4.0)\n", - "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->transformers==4.21) (3.0.6)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (2021.10.8)\n", - "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (1.26.8)\n", - "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (3.1)\n", - "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests->transformers==4.21) (2.0.7)\n", - "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.7.2)\n", - "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (4.0.1)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.2.0)\n", - "Requirement already satisfied: attrs>=17.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (21.2.0)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (5.2.0)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from aiohttp->datasets) (1.2.0)\n", - "Requirement already satisfied: python-dateutil>=2.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas->datasets) (2.8.2)\n", - "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas->datasets) (2021.3)\n", - "Requirement already satisfied: six>=1.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n", - "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.\n", - "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n", - "\u001b[0m" - ] - } - ], + "outputs": [], "source": [ "!pip install \"transformers==4.21\" datasets --upgrade" ] }, { "cell_type": "code", - "execution_count": 3, - "id": "4c979dbd", + "execution_count": null, + "id": "39b17a6f", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.4\n", - " warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "sagemaker: 2.112.2\n", - "transformers: 4.21.0\n" - ] - } - ], + "outputs": [], "source": [ "import botocore\n", "import boto3\n", @@ -409,7 +108,7 @@ }, { "cell_type": "markdown", - "id": "52fad7c5", + "id": "cc38670d", "metadata": {}, "source": [ "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for datasets library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", @@ -425,7 +124,7 @@ }, { "cell_type": "markdown", - "id": "710cf271", + "id": "68064ca2", "metadata": {}, "source": [ "### SageMaker Environment " @@ -433,20 +132,10 @@ }, { "cell_type": "code", - "execution_count": 4, - "id": "adbc14fb", + "execution_count": null, + "id": "f841807d", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "sagemaker role arn: arn:aws:iam::875423407011:role/AdminRole\n", - "sagemaker bucket: sagemaker-us-west-2-875423407011\n", - "sagemaker session region: us-west-2\n" - ] - } - ], + "outputs": [], "source": [ "import sagemaker\n", "\n", @@ -469,7 +158,7 @@ }, { "cell_type": "markdown", - "id": "b021041c", + "id": "c0d249c6", "metadata": {}, "source": [ "## SageMaker Training Job\n", @@ -481,14 +170,14 @@ }, { "cell_type": "code", - "execution_count": 5, - "id": "c874d8a8", + "execution_count": null, + "id": "32ef6ab0", "metadata": {}, "outputs": [], "source": [ "# Here we configure the training job. Please configure the appropriate options below:\n", "\n", - "EPOCHS = 1000\n", + "EPOCHS = 800\n", "\n", "# Choose between Causal Language Model and Masked Language Model\n", "LANGUAGE_MODELING_LOSS = \"clm\" # or \"mlm\"\n", @@ -510,7 +199,7 @@ }, { "cell_type": "markdown", - "id": "6431fe3c", + "id": "a7935fae", "metadata": {}, "source": [ "First, we define some basic parameters common to all estimators.\n", @@ -520,8 +209,8 @@ }, { "cell_type": "code", - "execution_count": 6, - "id": "5ee306d6", + "execution_count": null, + "id": "d3402d03", "metadata": {}, "outputs": [], "source": [ @@ -553,7 +242,7 @@ }, { "cell_type": "markdown", - "id": "d03ac353", + "id": "eded57be", "metadata": {}, "source": [ "Next, we define some basic arguments to be passed to the training script." @@ -561,8 +250,8 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "33960dc2", + "execution_count": null, + "id": "4430ec63", "metadata": {}, "outputs": [], "source": [ @@ -574,8 +263,6 @@ " \"dataset_config_name\": \"sst2\",\n", " \"do_train\": True,\n", " \"do_eval\": False,\n", - " \"fp16\": True,\n", - " \"per_device_train_batch_size\": 32,\n", " \"num_train_epochs\": EPOCHS,\n", " \"overwrite_output_dir\": True,\n", " \"evaluation_strategy\": \"no\",\n", @@ -589,7 +276,7 @@ }, { "cell_type": "markdown", - "id": "e00a9431", + "id": "63a0ad37", "metadata": {}, "source": [ "In the following sections, we will create estimators and start training." @@ -597,7 +284,7 @@ }, { "cell_type": "markdown", - "id": "42a0e4e7", + "id": "ce809ad9", "metadata": {}, "source": [ "### Training with Native PyTorch + SM DDP" @@ -605,7 +292,7 @@ }, { "cell_type": "markdown", - "id": "32039258", + "id": "8fa6201d", "metadata": {}, "source": [ "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training.\n", @@ -615,24 +302,15 @@ }, { "cell_type": "code", - "execution_count": 8, - "id": "98391f00", + "execution_count": null, + "id": "ff8e4608", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'trcomp-pt-example-2022-10-13-20-02-06-586'" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "from sagemaker.pytorch import PyTorch\n", "\n", + "hyperparameters[\"per_device_train_batch_size\"] = 16\n", + "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", " float(\"5e-5\") / 32 * hyperparameters[\"per_device_train_batch_size\"]\n", @@ -655,7 +333,7 @@ }, { "cell_type": "markdown", - "id": "64ff8e38", + "id": "adc1af25", "metadata": {}, "source": [ "### Training with SageMaker Training Compiler " @@ -663,7 +341,7 @@ }, { "cell_type": "markdown", - "id": "10108936", + "id": "3a4e5ef6", "metadata": {}, "source": [ "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately." @@ -671,26 +349,15 @@ }, { "cell_type": "code", - "execution_count": 9, - "id": "afd6d9d7", + "execution_count": null, + "id": "be0e51d0", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'trcomp-pt-example-2022-10-13-20-02-07-185'" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", "\n", "# with SageMaker Training Compiler we are able to fit a larger batch into memory\n", - "hyperparameters[\"per_device_train_batch_size\"] = 64\n", + "hyperparameters[\"per_device_train_batch_size\"] = 32\n", "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", @@ -714,7 +381,7 @@ }, { "cell_type": "markdown", - "id": "8c95d3f9", + "id": "46b6f94f", "metadata": {}, "source": [ "### Wait for training jobs to complete\n" @@ -722,8 +389,8 @@ }, { "cell_type": "code", - "execution_count": 23, - "id": "f6a9d068", + "execution_count": null, + "id": "a9f1e382", "metadata": {}, "outputs": [], "source": [ @@ -739,7 +406,7 @@ }, { "cell_type": "markdown", - "id": "f08e5948", + "id": "12e0aa73", "metadata": {}, "source": [ "## Analysis" @@ -747,7 +414,7 @@ }, { "cell_type": "markdown", - "id": "ab81168f", + "id": "20c2a1eb", "metadata": {}, "source": [ "**Note:** If the estimator object is no longer available due to a kernel break or refresh, you need to directly use the training job name and manually attach the training job to a new HuggingFace estimator. For example:\n", @@ -759,7 +426,7 @@ }, { "cell_type": "markdown", - "id": "a54ac611", + "id": "cb489539", "metadata": {}, "source": [ "### Load logs of the training job *with* SageMaker Training Compiler" @@ -767,8 +434,8 @@ }, { "cell_type": "code", - "execution_count": 24, - "id": "26da5643", + "execution_count": null, + "id": "0eac1589", "metadata": {}, "outputs": [], "source": [ @@ -780,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "8245afb8", + "id": "b103b5ce", "metadata": {}, "source": [ "### Load logs of the training job *without* SageMaker Training Compiler" @@ -788,8 +455,8 @@ }, { "cell_type": "code", - "execution_count": 25, - "id": "f57c7a57", + "execution_count": null, + "id": "8e07e595", "metadata": {}, "outputs": [], "source": [ @@ -801,7 +468,7 @@ }, { "cell_type": "markdown", - "id": "1dbc9558", + "id": "5a64aa04", "metadata": {}, "source": [ "### Create helper functions for analysis" @@ -809,8 +476,8 @@ }, { "cell_type": "code", - "execution_count": 26, - "id": "79b199de", + "execution_count": null, + "id": "12b72ee5", "metadata": {}, "outputs": [], "source": [ @@ -851,7 +518,7 @@ }, { "cell_type": "markdown", - "id": "65ceecc4", + "id": "7db33694", "metadata": {}, "source": [ "### Plot Optimized vs Native Training Throughput\n", @@ -861,8 +528,8 @@ }, { "cell_type": "code", - "execution_count": 27, - "id": "2382157d", + "execution_count": null, + "id": "85dcaaf7", "metadata": { "scrolled": true }, @@ -882,35 +549,10 @@ }, { "cell_type": "code", - "execution_count": 28, - "id": "3d71417c", + "execution_count": null, + "id": "aa270a8c", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "([,\n", - " ],\n", - " [Text(1.0, 0, 'PT + SM DDP'), Text(1.5, 0, 'PT + Compiler')])" - ] - }, - "execution_count": 28, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAElCAYAAAAV9s4VAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkbElEQVR4nO3deZhcZZXH8e+PsGgQBEzLkgSCEFRgMEBkccGgDgZhBGcUiIyAgwZGGHXGUQEZQCSyKOIgDMgaUFYFBDHKJgIuEZoQSVgCYTMJAZo1bAYSzvzxvgU3RVXfqqSrqzr9+zxPPXXve7dzO5U69d7lXEUEZmZmvVmh3QGYmVnnc7IwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYQOepN9I2rev5+1Lkh6W9PH+3m4tkkLSxu2OwwaWFdsdgA1Okl4ojA4FFgKL8/gBEXFBo+uKiJ1bMW+jJO0N/CSPDgFWAV4qbPNtfb3NTiRpFPAQsFJELGpzONbH3LOwtoiIt1VewN+Afyq0vZ4oJHX8D5qIuKCwLzsDj1btX1MGwj7b4ONkYR1F0jhJcyV9S9JjwLmS1pR0taQeSc/k4RGFZX4v6Yt5eD9Jf5D0gzzvQ5J2Xsp5N5R0s6TnJV0v6VRJP1uG3Rsj6U5Jz0m6RNJbetnnVST9SNKj+fUjSasU4676u71+aEnSOyT9StICSbdJOqZ6fuDjku6X9GzeLxXW/UdJp+Q475X0scJ2ljicJumowt/k5vz+rKQXJG2/DH8r6zBOFtaJ1gHWAjYAJpI+p+fm8fWBl4FTell+W2AWMAw4ATi78mXY5LwXArcC7wCOAj6/1HuU7AGMBzYEtgD2K0yr3udvA9sBY4D3AdsAhze4nVOBF/M6982varsC789x7AF8ojBtW+AB0t/kSOBySWs1sN0d8vsauVf15wbjtQHAycI60WvAkRGxMCJejoinIuKyiHgpIp4HJgEf6WX5RyLizIhYDJwHrAus3cy8ktYnfZkeERGvRMQfgKuWcb9OjohHI+Jp4FekRFCxxD4DewNHR8QTEdEDfIcGkpWkIcC/5HW9FBF35/2qdlxEPBsRfwNurIrlCeBHEfFqRFxCSqa7NLuztnxxsrBO1BMRf6+MSBoq6SeSHpG0gHS4Y438xVjLY5WBiKicaK537qDevOsBTxfaAOY0uR91t0U6AV6MaYl9ztt/pDD+SG4r00W6cKUYa624e4tlXixZYbTRbdtyzMnCOlF1KeSvA+8Gto2I1XnjcEe9Q0t9YT6wlqShhbaRLdxe9T4/SjokVbF+boN0iOn1uCStU5ivB1gEjCi0NRv38KrDdnW3TTrUVeES1ssxJwsbCFYjnad4Nh87P7LVG4yIR4Bu4ChJK+eTtf/U6u0WXAQcLqlL0jDgCKByIvmvwGaSxuST5EcV4l4MXJ7jHirpPcA+TW77ncBXJK0k6bPAe4Epedp0YK88bSzwmcJyPaTDae9qcns2ADhZ2EDwI+CtwJPAVOC3/bTdvYHtgaeAY4BLSPeD9IdjSMnqTmAGMC23ERH3AUcD1wP3A9VXOh0MvJ10qOmnpMTTTNx/AUaT/t6TgM9ExFN52v8AGwHPkM6jXFhZKB+ymwT8MV9ltV0T27QOJz/8yKwxki4B7o2Ilvds+pKk44F1IqL0znVJ+wFfjIgPtTwwG1DcszCrQ9L7JW0kaQVJ44HdgF+2OaxSkt4jaQsl2wD7A1e0Oy4b2HynqFl965CO/78DmAv8e0Tc0d6QGrIa6dDTesDjwInAlW2NyAY8H4YyM7NSPgxlZmalnCxswJF0kaTdm5h/b0nXNjjvm+ou9SVJh0k6aymW65gS5wC5dtW9krraHYv1DycLG1AkbUGqlXRlHq/55V78cs1VYXfq30hri4jvRcQX+3ObNYr/jcqFB5f6nGVELATOAQ7pixit8zlZ2EBzAHBBDICTbcvyZdzJCvt1IbBvpRquLd+cLGyg2Rm4qZkFqnsfknaSNCuX4P4/STcply0vzFOvbPnbJZ0tab6kebn895DCdv4o6SRJT1G4s7qw/OslvSW9RdLPJD2Vb2K7TVK9gocA75d0d47r3Hz3dmW9u0qantfzp9wDQ9JPSeU6fpXLhn+TOqXEJf2bpHvy+q+RtEFh/SHpIEn3k24EJCLmkm7O8813g4CThQ0YklYllfeetQzrGAb8AjiUdEnsLOADVbP1VrZ8Mqn20sbAlsBOwBerln2QVOV2Ukk4+5LutB6ZYzmQVNaknr1JpcQ3AjYhlyyXtCXpkNABeT0/Aa6StEpEfJ4lHy51AjVKiUvaDTgM+GdSMcJbSJffFu2e92/TQts9pMOCtpxzsrCBZI38/nxV+3b5F/XrL9Kv6Vo+CdwVEZfnR3+ezJIVWKF+2fK18/Jfi4gXI+IJ4CRgr8Kyj0bEjyNiUS413ptXSV/uG0fE4oi4PSIW9DL/KRExJ5c4nwRMyO0TgZ9ExF/yes4jlfdo5hf/gcCxEXFP/rt8j/SwpmIxw2Mj4umq/XqeN/5dbDnmZGEDybP5fbWq9qkRsUbxRfo1Xct6FEp253Mfc6vmqVe2fANgJWB+ISn9hFR4r6KZMuY/Ba4BLlZ6Gt4JklbqZf7iuotlwzcAvl6VLEfSXFnxDYD/LSz/NKmq7/A6269YjTf+XWw55mRhA0ZEvEh6gtsmy7Ca+RTKd+fDSyPqz76EOaRf7MMKiWn1iNisGGajgeSHC30nIjYlHQrbld4rxBZLjRfLhs8BJlUlzKERUTmMVB1TrRjnAAdUreOtEfGnkuXeS6qCa8s5JwsbaKbQ+1Pyyvwa+AdJu+ereg5iyWcy1BUR84FrgRMlrZ5rRm0kaanikbSjpH/IJ8gXkA5LvdbLIgdJGqFUpv3bpCq4AGcCB0raNteDWlXSLpIqPbDHWbJseK1S4qcDh0raLMf2dqXy5L3FP5z0KNipDe2wDWhOFjbQnAHsXTjh3JSIeBL4LOnE9VOkk7XdNF7Cex9gZeBu0pVAvyCd01ga6+TlF5BOFN9EOjRVz4WkZPUgqYdVKVneDXyJ9FzyZ4DZLPl872NJz8Z4VtJ/1yolHhFXAMeTDoktAGaSrjzrzeeA8/I9F7acc20oG3AkXQhcGhG/7IN1rUA6Z7F3RNy4rOsbLPK9FX8Fdsgn+m0552Rhg46kT5Ae8PMy8A3Soah3NXD1ktmg5cNQNhhtTzqM8yTpUam7O1GY9c49CzMzK+WehZmZlVouC50BDBs2LEaNGtXuMMzMBozbb7/9yYioWXZ+uU0Wo0aNoru7u91hmJkNGJIeqTfNh6HMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKzUcnsHt5kBR7293RFYfzvquZas1j0LMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZqZYlC0nnSHpC0sxC2yWSpufXw5Km5/ZRkl4uTDu9sMzWkmZImi3pZElqVcxmZlZbK+/gngycApxfaYiIPSvDkk4EircaPhARY2qs5zTgS8BfgCnAeOA3fR+umZnV07KeRUTcDDxda1ruHewBXNTbOiStC6weEVMjIkiJZ/c+DtXMzEq065zFh4HHI+L+QtuGku6QdJOkD+e24cDcwjxzc1tNkiZK6pbU3dPT0/dRm5kNUu1KFhNYslcxH1g/IrYE/gu4UNLqza40Is6IiLERMbarq6uPQjUzs36vOitpReCfga0rbRGxEFiYh2+X9ACwCTAPGFFYfERuMzOzftSOnsXHgXsj4vXDS5K6JA3Jw+8CRgMPRsR8YIGk7fJ5jn2AK9sQs5nZoNbKS2cvAv4MvFvSXEn750l78eYT2zsAd+ZLaX8BHBgRlZPjXwbOAmYDD+AroczM+l3LDkNFxIQ67fvVaLsMuKzO/N3A5n0anJmZNcV3cJuZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpVqWLCSdI+kJSTMLbUdJmidpen59sjDtUEmzJc2S9IlC+/jcNlvSIa2K18zM6mtlz2IyML5G+0kRMSa/pgBI2hTYC9gsL/N/koZIGgKcCuwMbApMyPOamVk/WrFVK46ImyWNanD23YCLI2Ih8JCk2cA2edrsiHgQQNLFed67+zpeMzOrrx3nLA6WdGc+TLVmbhsOzCnMMze31WuvSdJESd2Sunt6evo6bjOzQau/k8VpwEbAGGA+cGJfrjwizoiIsRExtqurqy9XbWY2qLXsMFQtEfF4ZVjSmcDVeXQeMLIw64jcRi/tZmbWT/q1ZyFp3cLop4HKlVJXAXtJWkXShsBo4FbgNmC0pA0lrUw6CX5Vf8ZsZmYt7FlIuggYBwyTNBc4EhgnaQwQwMPAAQARcZekS0knrhcBB0XE4ryeg4FrgCHAORFxV6tiNjOz2lp5NdSEGs1n9zL/JGBSjfYpwJQ+DM3MzJrkO7jNzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1ItSxaSzpH0hKSZhbbvS7pX0p2SrpC0Rm4fJellSdPz6/TCMltLmiFptqSTJalVMZuZWW2t7FlMBsZXtV0HbB4RWwD3AYcWpj0QEWPy68BC+2nAl4DR+VW9TjMza7GWJYuIuBl4uqrt2ohYlEenAiN6W4ekdYHVI2JqRARwPrB7C8I1M7NetPOcxb8BvymMbyjpDkk3SfpwbhsOzC3MMze31SRpoqRuSd09PT19H7GZ2SDVULKQtKqkFQrjK0gaurQblfRtYBFwQW6aD6wfEVsC/wVcKGn1ZtcbEWdExNiIGNvV1bW04ZmZWZVGexY3AMXkMBS4fmk2KGk/YFdg73xoiYhYGBFP5eHbgQeATYB5LHmoakRuMzOzftRosnhLRLxQGcnDTfcsJI0Hvgl8KiJeKrR3SRqSh99FOpH9YETMBxZI2i5fBbUPcGWz2zUzs2WzYoPzvShpq4iYBulyVuDl3haQdBEwDhgmaS5wJOnqp1WA6/IVsFPzlU87AEdLehV4DTgwIionx79MurLqraRzHMXzHC0x6pBft3oT1mEePm6Xdodg1tEaTRZfA34u6VFAwDrAnr0tEBETajSfXWfey4DL6kzrBjZvME4zM2uBhpJFRNwm6T3Au3PTrIh4tXVhmZlZJ2n0aqihwLeAr0bETGCUpF1bGpmZmXWMRk9wnwu8Amyfx+cBx7QkIjMz6ziNJouNIuIE4FWAfCWTazSZmQ0SjSaLVyS9FQgASRsBC1sWlZmZdZRGr4Y6EvgtMFLSBcAHgf1aFZSZmXWWRq+Guk7SNGA70uGnr0bEky2NzMzMOkajV0N9EPh7RPwaWAM4TNIGrQzMzMw6R6PnLE4DXpL0PlKhvwdI5cLNzGwQaDRZLMpF/3YDTo2IU4HVWheWmZl1kkZPcD8v6VDgX4EdcrnylVoXlpmZdZJGexZ7ki6V3T8iHiOVCv9+y6IyM7OO0mvPQtI1pEtmfxMRP6y0R8Tf8DkLM7NBo6xnsS/wDHCUpGmSTpO0m6RV+yE2MzPrEL32LPIhp8nA5HyeYltgZ+Cbkl4Grs1lQMzMbDnW6AluIuI14M/5dYSkYcAnWhWYmZl1jkZvyjtB0uqSVpJ0g6QeYHxEXNDi+MzMrAM0ejXUThGxANgVeBjYGPhGq4IyM7PO0miyqByu2gX4eUQ818hCks6R9ISkmYW2tSRdJ+n+/L5mbpekkyXNlnSnpK0Ky+yb579f0r4NxmxmZn2k0WRxtaR7ga2BGyR1AX9vYLnJwPiqtkOAGyJiNHBDHod04nx0fk0klRhB0lqkqrfbAtsAR1YSjJmZ9Y+GkkVEHAJ8ABibn739Eqn0R9lyNwNPVzXvBpyXh88Ddi+0nx/JVGANSeuSTqJfFxFPR8QzwHW8OQGZmVkLNfMM7i+Tf+0D6wFjl3Kba0fE/Dz8GLB2Hh4OzCnMNze31Ws3M7N+0uwzuD+Qx/vkGdy5OGEs63oqJE2U1C2pu6enp69Wa2Y26LXjGdyP58NL5Pcncvs8YGRhvhG5rV77m0TEGRExNiLGdnV1LWV4ZmZWrR3P4L6KVEaE/H5loX2ffFXUdsBz+XDVNcBOktbMJ7Z3ym1mZtZPWvoMbkkXAeOAYZLm5vUcB1wqaX/gEWCPPPsU4JPAbNIJ9C8ARMTTkr4L3JbnOzoiqk+am5lZC7X0GdwRMaHOpI/VmDeAg+qs5xzgnEZiNTOzvldWonyrqqbKVUzrS1o/Iqa1JiwzM+skZT2LE3uZFsBH+zAWMzPrUGUlynfsr0DMzKxzNXTOQtJbSDflfYjUo7gFOD0iGin5YWZmA1yjV0OdDzwP/DiPfw74KfDZVgRlZmadpdFksXlEbFoYv1HS3a0IyMzMOk+jN+VNyzfKASBpW6C7NSGZmVmnabRnsTXwJ0l/y+PrA7MkzSDdIrFFS6IzM7OO0GiycElwM7NBrNE7uB/JdZlGFpfxTXlmZoNDo5fOfpdUC+oB3igp7pvyzMwGiUYPQ+1BKlP+SiuDMTOzztTo1VAzgTVaGIeZmXWwRnsWxwJ3SJpJ4TkWEfGplkRlZmYdpdFkcR5wPDADeK114ZiZWSdqNFm8FBEntzQSMzPrWI0mi1skHUt69GnxMJQvnTUzGwQaTRZb5vftCm2+dNbMbJBo9KY8P9fCzGwQa7RngaRdgM2At1TaIuLoZjco6d3AJYWmdwFHkC7N/RLQk9sPi4gpeZlDgf2BxcBXIuKaZrdrZmZLr9E7uE8HhgI7AmcBnwFuXZoNRsQsYExe7xBgHnAF8AXgpIj4QdW2NwX2IiWq9YDrJW0SEYuXZvtmZta8Rm/K+0BE7AM8ExHfAbYHNumD7X8MeCAiHullnt2AiyNiYUQ8BMwGtumDbZuZWYMaTRYv5/eXJK0HLALW7YPt7wVcVBg/WNKdks7JhQsBhgNzCvPMzW1vImmipG5J3T09PbVmMTOzpdBosrha0hrACcDtwEMs+SXfNEkrA58Cfp6bTgM2Ih2img+c2Ow6I+KMiBgbEWO7urqWJTwzMyvo9ZyFpPcDcyLiu3n8baS7uO8FTlrGbe8MTIuIxwEq73k7ZwJX59F5pNLoFSNym5mZ9ZOynsVPgFcAJO0AHJfbngPOWMZtT6DQO5FUPKz1aVLxQkg3Au4laRVJGwKjWcqT62ZmtnTKroYaEhFP5+E9gTMi4jLgMknTl3ajklYF/hE4oNB8gqQxpJv9Hq5Mi4i7JF0K3E06V3KQr4QyM+tfpclC0ooRsYh05dLEJpatKyJeBN5R1fb5XuafBExa2u2ZmdmyKfvCvwi4SdKTpCuibgGQtDHpUJSZmQ0CvSaLiJgk6QbSZbLXRkTlkaorAP/R6uDMzKwzlB5KioipNdrua004ZmbWiRq9z8LMzAYxJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKtS1ZSHpY0gxJ0yV157a1JF0n6f78vmZul6STJc2WdKekrdoVt5nZYNTunsWOETEmIsbm8UOAGyJiNHBDHgfYGRidXxOB0/o9UjOzQazdyaLabsB5efg8YPdC+/mRTAXWkLRuG+IzMxuU2pksArhW0u2SJua2tSNifh5+DFg7Dw8H5hSWnZvbliBpoqRuSd09PT2titvMbNBZsY3b/lBEzJP0TuA6SfcWJ0ZESIpmVhgRZwBnAIwdO7apZc3MrL629SwiYl5+fwK4AtgGeLxyeCm/P5FnnweMLCw+IreZmVk/aEuykLSqpNUqw8BOwEzgKmDfPNu+wJV5+Cpgn3xV1HbAc4XDVWZm1mLtOgy1NnCFpEoMF0bEbyXdBlwqaX/gEWCPPP8U4JPAbOAl4Av9H7KZ2eDVlmQREQ8C76vR/hTwsRrtARzUD6GZmVkNnXbprJmZdSAnCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpfo9WUgaKelGSXdLukvSV3P7UZLmSZqeX58sLHOopNmSZkn6RH/HbGY22K3Yhm0uAr4eEdMkrQbcLum6PO2kiPhBcWZJmwJ7AZsB6wHXS9okIhb3a9RmZoNYv/csImJ+REzLw88D9wDDe1lkN+DiiFgYEQ8Bs4FtWh+pmZlVtPWchaRRwJbAX3LTwZLulHSOpDVz23BgTmGxudRJLpImSuqW1N3T09OqsM3MBp22JQtJbwMuA74WEQuA04CNgDHAfODEZtcZEWdExNiIGNvV1dWX4ZqZDWptSRaSViIligsi4nKAiHg8IhZHxGvAmbxxqGkeMLKw+IjcZmZm/aQdV0MJOBu4JyJ+WGhftzDbp4GZefgqYC9Jq0jaEBgN3Npf8ZqZWXuuhvog8HlghqTpue0wYIKkMUAADwMHAETEXZIuBe4mXUl1kK+EMjPrX/2eLCLiD4BqTJrSyzKTgEktC8rMzHrlO7jNzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1IDJllIGi9plqTZkg5pdzxmZoPJgEgWkoYApwI7A5sCEyRt2t6ozMwGjwGRLIBtgNkR8WBEvAJcDOzW5pjMzAaNFdsdQIOGA3MK43OBbatnkjQRmJhHX5A0qx9iW54MA55sdxDtoOPbHYH1sUH7WeY7WpalN6g3YaAki4ZExBnAGe2OY6CS1B0RY9sdh9my8me57w2Uw1DzgJGF8RG5zczM+sFASRa3AaMlbShpZWAv4Ko2x2RmNmgMiMNQEbFI0sHANcAQ4JyIuKvNYS2PfAjPlhf+LPcxRUS7YzAzsw43UA5DmZlZGzlZmJlZKSeLDiJpsaTpkmZK+rmk4Xl8uqTHJM0rjK+8DNtZW9LVkv4q6W5JU3L7KEkh6ZjCvMMkvSrplBrr2U9Sj6Q7JN0v6RpJHyhMnyzpobyd+ySdL2lEYfrDkmZIulPStZLWWdp9ss7QX5/hvK2dJXXnz/Adkk7sq/2os72jJX08D/9e0qC6NNfJorO8HBFjImJz4BVgzzw+BjgdOKkynu9kf5P8hf/7ku0cDVwXEe+LiE2BYq2th4BdCuOfBXq7mOCSiNgyIkYDxwGXS3pvYfo3IuJ9wLuBO4DfVX1J7BgRWwDdwGElcVvn65fPsKTNgVOAf82f4bHA7L7ckWoRcUREXL80y+aSRQOak0XnugXYuEXrXpd0FzwAEXFnYdpLwD2FX017Apc2stKIuJF0FcrEGtMiIk4CHiPV+Kp2M63bX2uPVn6GvwlMioh7ASJicUScBq8nm9/lHusNktbP7ZMlnSZpqqQHJY2TdI6keyRNrqxY0guSTpJ0V16+q7D8Z6oDkbSTpD9LmpZ7U2/L7Q9LOl7SNNKPrgHNyaIDSVqR9IU6o0WbOBU4W9KNkr4tab2q6RcDe0kaCSwGHm1i3dOA9yzF9F1p3f5aP+uHz/DmwO11pv0YOC/3WC8ATi5MWxPYHvhP0r1aJwGbAf8gaUyeZ1WgOyI2A24CjqwXhKRhwOHAxyNiK1IP+b8KszwVEVtFxMXN7V7nGRD3WQwib5U0PQ/fApzd6IKSrgA2BFYG1i+s538j4tzivBFxjaR3AeNJ/6HvyN36it8C3wUeBy5pch/KCtNUT79R0mLgTtJ/OhvY+uUzXGJ74J/z8E+BEwrTfhURIWkG8HhEzMjbvgsYBUwHXuONz/3PgMt72dZ2pErYf5REjv3PhenN/v/pWE4WneXlfGy3aRHxaUhdcGByRIwrmf9p4ELgQklXAzuQf6lFxCuSbge+TvqP8KkmQtkSuKdk+g2F8R0jYnAWfFs+9ddn+C5ga+CvTW5mYX5/rTBcGa/3fdjbzWginf+bUGf6i82F17l8GGoQkvRRSUPz8GrARsDfqmY7EfhWTiqNrvcjpPMVZ9aYJklfIZ0v+e3Sxm6WfR84TNImAJJWkHRgnvYnUkkggL1JPZxmrABUzk18DvhDL/NOBT4oaeMcx6qVmJY37lkMTlsDp0haRPqPcVZE3JZ/0QGQy6k0UlJlT0kfAoaSrqT6l4go9iy+L+l/8vSppJ5EzatgzBoVEXdK+hpwUf7hE8DVefJ/AOdK+gbQA3yhydW/CGwj6XDgCdJFHvXi6JG0X45jldx8OHBfk9vseC73YWZWIOmFiHhbu+PoND4MZWZmpdyzMDOzUu5ZmJlZKScLMzMr5WRhZmalnCys4+UKn7MK1UrfmdsPzFVrp0v6g6RNc/sHc12gbkmjc9saubJtzc+8pJUkHadUPXdarvVTq4ZVX+7Xn/L7KEkzW7mtBuN5vaqqWTXfZ2H9TtKqwCsR8WoTi+0dEd1VbRdGxOl5nZ8CfkgqYfJ14JOk8g0H5vHDge9FxGt11v9d0g2Dm0fEQklrAx9pIr6mRcQHyud6M0krRsSiFsRzRF+vsxGS1mrm5k9rD/csrB02Ae6T9IOqcuZNiYgFhdFVeaMsw6ukmwCHAq9K2ggYGRG/r7WefFPXl4D/iIiFed2PR8SlefqE3IOZKen4wnIvSPp+rk56vaRtci/owZy8Ks/8uDK33y/pyOLyNWIZktd5W+4dHZDbx0m6RdJVwN01lpmc45sh6T9z+5fyev4q6TJJQyW9XdIjlR5WvuN4Tu5ZvV5VVali6ndyL2uGpPfk9i5J1+V9Piuva1hez6/ztmZKqnsjW17P6pIOkHQr8N+9zWudwT2LQUbSScCONSZdHBHHSdqRVImz2kuVX8KSbgFWqzHPf0fE9fnO2b1rTL85Ir4SEXdI2oJ0Z+xZkoJUcO7SiKhXS+dcpYKDlwHHRL7mW9JBpCqfKwMfzfMeC5wPvAx8HvgBvRcp3Bj4W1XyIa9/PeB40l3vzwDXSto9In5JSlC/i4hvKBXBOwb4R1I9rfNIVU0BtiFVSX0JuE3Sr2v0kir2B56LiPfnO4L/KOnaPG0rUs/noaplxgDD8zMkkLRGbr88Is7MbccA+0fEj5UK9H0EuJFU7feaiHhVelMNyCcjYitJXyZ9oX+RVIH1dxFxrKTxOV5IPbpHI2KXvL2319o5pbv9vwh8kPRv+a8RcV+e1vLPnqR3U7+437iIeLbONIsIv/xq6wt4L6mez4I604fn99WAa4F9aszzOVJZ6ur2HUhfQJuQviR+BqxdNc8WwB11tr0bcH5hfH/gh3l4IW/cq3Q08O08vALwbB7er2r5o4Gv5eEX8vsoYGYe/gWpVMT0/HoI2AkYB9xYJ8Y1gQdIpbnHAyvk9o+Q6iLNyOs5vfC3qgxfAfxjHp4MfCYPP1z4u28LXJ+HpwMbFrb9NDAs/30fJiXWD9eJ82RSwv0cMKTdnzu/mnu5ZzHIdELPohDLKGBfYAKpeuhRtWKOiHn5/XlJF5J+qZ9fHT9wWrFB6afy4aSicj8mPTBnFPAV4NuFWWeTSmKvHjV6F714NfK3IIUqphHxmtLzHF7fhepd6mWdIh0Ou6ZqX8ZRp4JpRDwj6X3AJ0jnaPYA/o305b97RPxVqX7RuLzIVcD3JK1F6jH9rk4slaqsiyk5ChER90nainSu6BhJN0TE0VWz/RBYQOqdjJd0LvD7yt/QPYsO1+5s5dfge5G+sK8n/Ur9KvCOXuZdERiWh1ci/fI+MI+PLsz3T6QH1hSX3Zc3fsVfAYwEPkx6tGf1dk4AzgVWzuNdpKebrQs8Qvr1PCTHvVue54XC8keRvrAoTiP1LB4F1gLeSnpux9iqeUbxRs9iIvBLYKU8vgnpcNc44Oo6f6NhwOp5eHNgeh5+Enhn/rtdRyr7XVnm56RnPfxfoW0yS/YsKn/3saQvdUgPzvpWHt6JlPiGAesBb8ntuwK/7OXfdAgpqVwOzCJdvND2z6Vfvb/cs7B2WAwcFhG3NjDvKsA1klbijS/rSgn0g/Olnq+SDm/sW1kon7Tej/SFBulX7RTSc6E/V2M7h5POOdwt6e+kX/FHRMR8SYeQju8L+HVEXNnEvgLcSjo+PwL4WdQ/XwFwFil5TMs9ox5g95L1Dyed06lcsHJofv8f4C95HX9hyV/kl5ASxrhGdyL7DqnC6udJD/l5DHg+r+f7kl4j/Xv8e70VRMRi0r/FFKXLoJfLkt7LG9eGMmuhfPhnbEQc3O5Y+kI+6b44IhZJ2h44LZbyYUc2sLhnYWbNWB+4NPdiXiFdcmyDgHsWZmZWyjflmZlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZX6f0xAIcTawUzpAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -926,7 +568,7 @@ }, { "cell_type": "markdown", - "id": "d1713454", + "id": "db11ce58", "metadata": {}, "source": [ "### Convergence of Training Loss\n", @@ -936,33 +578,10 @@ }, { "cell_type": "code", - "execution_count": 29, - "id": "b02abff1", + "execution_count": null, + "id": "5bc8a723", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5z0lEQVR4nO3deXxU9bn48c8zSSb7npCFEBJ2CEtYRBBxAQXrrtUqLlVrf7a2ttbrtbW3Xlt7W6/tba22dtFWq3Wl7orFHbQggiBbCCBbAgmBLITs68z398c5CQGTkG0yk5nn/Xqd18ycc+ac52TgOed8tyPGGJRSSgUOh7cDUEopNbg08SulVIDRxK+UUgFGE79SSgUYTfxKKRVgNPErpVSA0cSvBpWIrBSRbw7Svm4VkcMiUisiiR7cT62IjBrodZXyFE38asCJSIGINNhJ7rCIPCkiUb3cRpaIGBEJ7mMMIcCDwCJjTJQxpqLDsvl2bLUiUmfvp7bDlNmbfdnb3zvQ6/aGiPxMRJ4Z6O0q/6SJX3nKRcaYKGAGMAu4Z5D3nwKEAdtOXGCM+bedgKOAHHt2XNs8Y8z+tnX7euJRypdp4lceZYwpBpYDk09cJiIOEblHRApFpFRE/iEisfbij+3Xo/ZV+NxOvh8qIg+JyEF7esieNw7Y2eH7H/Y0XvvK+SUReUZEqoEbRWS2iKwRkaMiUiIij4iIs8N3jIiMsd8/KSJ/FJG3RKRGRNaKyOg+rrtIRHaKSJWI/ElEPupLMZmIXCwi2+z4V4rIxA7LfiQixfb+d4rIQnv+bBFZLyLV9l3bg73dr/JdmviVR4nICOB8YGMni2+0p7OBUUAU8Ii97Az7te1KfE0n3/8JMAfIBaYBs4F7jDFfcPyV/IJehn0J8BIQBzwLuIA7gCRgLrAQ+E43378auA+IB3YDv+ztuiKSZMfwYyAR60R2Wi+PA/sk+DzwAyAZ+Bfwpog4RWQ8cBtwijEmGlgMFNhffRh42BgTA4wG/tnbfSvfpYlfecprInIUWAV8BNzfyTrXAg8aY/YaY2qxktzVvSheuRb4uTGm1BhThpVAr+9/6KwxxrxmjHEbYxqMMRuMMZ8aY1qNMQXAo8CZ3Xz/VWPMOmNMK9aJI7cP654PbDPGvGIv+z1wqA/HchXwljHmPWNMC/AbIBzrJOICQoFJIhJijCkwxuyxv9cCjBGRJGNMrTHm0z7sW/koTfzKUy41xsQZY0YaY75jjGnoZJ10oLDD50IgGKt8vic6+356n6I93oGOH0RknIgsE5FDdvHP/VhX/13pmKDrse5kertuesc4jDWaYlEPYj/RcX8jY4zb3u5wY8xurDuBnwGlIvKCiLT9/W4GxgE7ROQzEbmwD/tWPkoTv/Kmg8DIDp8zgVbgMNCTYWM7+/7BAYjrxH3/GdgBjLWLPv4LkAHYT3dKgIy2DyIiHT/3wnF/I3s7I4BiAGPMc8aY0+11DPAre/4uY8wSYJg97yURiezboShfo4lfedPzwB0ikm0397wfWGoXbZQBbqyy/+6+f4+IJNtl4vcCnmjSGA1UA7UiMgG41QP7ONFbwBQRudQu+voukHqS7zhEJKzDFIpVNn+BiCy0m7jeCTQBn4jIeBFZYK/XCDRg/c0RketEJNm+Qzhqb9894EepvEITv/KmJ4CnsVrw7MNKPt8DMMbUY1V0rrZbo8zp5Pu/ANYDW4CtwOf2vIH2n8A1QA3wV2CpB/ZxHGNMOXAl8GugApiEdaxN3XxtCVbybpv2GGN2AtcBfwDKgYuwmto2Y5XvP2DPP4R1df9je1vnAdtEpBarovfqLorr1BAk+iAWpXyfiDiwyvivNcas8HY8amjTK36lfJSILBaROLsopq1eQVvXqH7TxK+U75oL7OFYEc2lWtyiBoIW9SilVIDRK36llAowQ2IAqqSkJJOVleXtMJRSakjZsGFDuTEm+cT5QyLxZ2VlsX79em+HoZRSQ4qIFHY2X4t6lFIqwGjiV0qpAKOJXymlAsyQKONXSg2+lpYWioqKaGxs9HYo6iTCwsLIyMggJCSkR+tr4ldKdaqoqIjo6GiysrKwBvVUvsgYQ0VFBUVFRWRnZ/foO1rUo5TqVGNjI4mJiZr0fZyIkJiY2Ks7M038SqkuadIfGnr7O/l34t/8Aqx/wttRKKWUT/HvxJ/3Mnz+D29HoZTqo6CgIHJzc5k8eTJXXnklxcXF5ObmkpubS2pqKsOHD2//3Nzc3Of9HD58mAsvvJBp06YxadIkzj//fAAKCgoQEe655572dcvLywkJCeG222770naefPJJkpOTmT59OmPHjmXx4sV88skn7ctvvPFGsrOzmTZtGuPGjePrX/86RUXHnqiZlZXFlClTmDp1KosWLeLQob48Zvnk/DvxO4LB3ertKJRSfRQeHs6mTZvIy8vD6XSydOlSNm3axKZNm/j2t7/NHXfc0f7Z6XR2uo2CggLOOuusbvdz7733cu6557J582by8/N54IEH2pdlZ2fz1ltvtX9+8cUXycnJ6XJbV111FRs3bmTXrl3cfffdXH755Wzfvr19+f/93/+xefNmdu7cyfTp01mwYMFxJ60VK1awZcsWZs2axf3333+yP1Gf+HfiFwe4Xd6OQik1AObPn8/u3bs9su2SkhIyMo490njq1Knt7yMiIpg4cWL7sDFLly7la1/7Wo+2e/bZZ3PLLbfw2GOPfWmZiHDHHXeQmprK8uXLv7T8jDPO8Njx+ndzTr3iV2pA3PfmNvIPVg/oNielx/DTi7q+cu6otbWV5cuXc9555w1oDG2++93vctVVV/HII49wzjnncNNNN5Gent6+/Oqrr+aFF14gJSWFoKAg0tPTOXjwYI+2PWPGDB599NFul+/YsYNLLrnkuPnLli1jypQpfTugkwiAxK9X/EoNVQ0NDeTm5gLWFf/NN9/c4+9edtll7Nu3j+bmZvbv39++ndtvv52bbrrpuHUXL17M3r17efvtt1m+fDnTp08nLy+vffl5553Hf//3f5OSksJVV13Vq2M42TNPTlx+9tlnExQUxNSpU/nFLzzxCOmASPx6xa9Uf/X0ynygtZXx98Wrr74KWGX8N954IytXrux2/YSEBK655hquueYaLrzwQj7++GNmzpwJgNPpZObMmfz2t78lPz+fN954o8dxbNy4kYkTJ3a7fOHChe2fV6xYQVJSUo+33xd+nviD9IpfKXVSH374IXPmzCEiIoKamhr27NlDZmbmcevceeednHnmmSQkJPR4ux999BGPPfYYK1as+NIyYwx/+MMfKCkp8VgRVlf8P/EbTfxKqe5t2LCB2267jeDgYNxuN9/85jc55ZRTKCgoaF8nJyen29Y8bZYuXcqqVauor68nOzubl19++bgr/rvuuov/+Z//ob6+njlz5rBixYouWyR5ypB45u6sWbNMnx7EsuwO2P4m3OWZmnGl/Nn27du7LaJQvqWz30tENhhjZp24rn8359QyfqWU+hL/TvyiZfxKKXUi/078WrmrlFJf4ueJX4t6lFLqRH6e+IM08Sul1An8OvHvKK23mnMOgZZLSik1WPw68W8/3GC9MW7vBqKU6pPBGpYZYPny5cyaNYtJkyYxffp07rzzzgE6is7de++9vP/++wCcddZZ9KnJeh95LPGLyBMiUioieR3mJYjIeyKyy36N99T+AYwEWW+0uEepIWmwhmXOy8vjtttu45lnniE/P5/169czZswYDxzRMT//+c8555xz+vRdl6t/jVY8ecX/JHBiP+S7gQ+MMWOBD+zPHiNBmviV8heeHJb517/+NT/5yU+YMGECYN1p3HrrrYB14liwYAFTp05l4cKF7N+/H7AeqnLrrbcyZ84cRo0axcqVK/nGN77BxIkTufHGG9u3HRUVxR133EFOTg4LFy6krKys/fsvvfTSl2J59913mTt3LjNmzODKK6+ktrYWsB7S8qMf/YgZM2bw4osv9ut4PTZkgzHmYxHJOmH2JcBZ9vungJXAjzwVgzjsw9MmnUr1z/K74dDWgd1m6hT4ygMnXw/PD8ucl5fXZdHO9773PW644QZuuOEGnnjiCb7//e/z2muvAVBZWcmaNWt44403uPjii1m9ejV/+9vfOOWUU9i0aRO5ubnU1dUxa9Ysfve73/Hzn/+c++67j0ceeaTTfZWXl/OLX/yC999/n8jISH71q1/x4IMPcu+99wKQmJjI559/3u/jHeyxelKMMSX2+0NASlcrisgtwC3AlwZL6rH2xK9X/EoNRYM1LHN31qxZwyuvvALA9ddfzw9/+MP2ZRdddBEiwpQpU0hJSWkfPz8nJ4eCggJyc3NxOBztQzlfd911XH755V3u69NPPyU/P5958+YB0NzczNy5c9uX93ZI6K54bZA2Y4wRkS6b2xhjHgMeA2usnr7sQxxtRT16xa9Uv/TwynygDdawzDk5OWzYsIFp06b1ah+hoaEAOByO9vdtn1tbO7/gFJEut2eM4dxzz+X555/vdHlkZGSv4uvKYLfqOSwiaQD2a6kndyZBdmWPq3+1/Uop/3bXXXdx//3388UXXwDgdrv5y1/+AsBpp53GCy+8AMCzzz7L/Pnze7Vtt9vdXpb/3HPPcfrpp3e57pw5c1i9enV7XUZdXV17TANpsBP/G8AN9vsbgNc9uTNXcLj1pqXek7tRSg1xU6dO5aGHHmLJkiVMnDiRyZMns3fvXgD+8Ic/8Pe//52pU6fy9NNP8/DDD/dq25GRkaxbt47Jkyfz4YcftpfXdyY5OZknn3ySJUuWMHXqVObOncuOHTv6dWyd8diwzCLyPFZFbhJwGPgp8BrwTyATKAS+Zow5crJt9XVY5peffZSv7voh3PIRpOf2+vtKBTIdlnlgREVFtbfM8aTeDMvsyVY9S7pYtLCL+QPOHRJhxdJcR9elakopFVj8uueusRO/u8nzZ1ullOrMYFzt91ZAJH5XU52XI1FqaBoKT+hTvf+d/Drx47SaPrkafe+Mq5SvCwsLo6KiQpO/jzPGUFFRQVhYWI+/49cPW28JS6LehBJUuApmf93b4Sg1pGRkZFBUVNQ+xIDyXWFhYWRkZPR4fb9O/EFhUWx0j2FW5T5vh6LUkBMSEkJ2dra3w1Ae4NdFPfERTiqJgvqTthhVSqmA4deJPyHSSaWJxtFQ4e1QlFLKZ/h54g/hCNEEN1WBSwdqU0op8PPEHx/hZLd7OIIbDm3xdjhKKeUT/Drxx0U42WxGWx8GeixxpZQaovw68Qc5hLqwNFrECRW7vB2OUkr5BL9O/ACxUeGUhgyHcs88sk0ppYYav0/8CRFOihzDoXzgx7RWSqmhyP8Tf6STPWY4VBZAS6O3w1FKKa8LiMSf1zocjEvL+ZVSigBI/PGRTjY2plofSrd7NxillPIBfp/4EyKc7HalYhwhUJrv7XCUUsrr/D/xRzppIZiW+NF6xa+UUgRI4geoixmrV/xKKUUAJP54O/EfiRwNR/eDPoZRKRXg/D7xJ0eHAlASmmXNKN/pvWCUUsoH+H/ij7IS/x7JtGaU7vBiNEop5X1+n/idwQ4SIp3sak6E4DAo0wpepVRg8/vEDzAsOpTDta2QNFZb9iilAl5gJP6YMEqrGyF5ohb1KKUCXkAk/pToUA5XN8GwCVBdBI3V3g5JKaW8JiAS/7CYUMpqm3AlTbBmlGnLHqVU4AqIxJ8SE4bLbTgaZT+NSyt4lVIBLCAS/7DoMABKJAWCw7WCVykV0AIj8cdYbflLa5shebwmfqVUQPNK4heRO0Rkm4jkicjzIhLmyf2lxFibL61ugmEToUxb9iilAtegJ34RGQ58H5hljJkMBAFXe3Kfbb13D1U3QvIEqCmBhqOe3KVSSvksbxX1BAPhIhIMRAAHPbkzZ7CDpCgnh6sbrSt+0Kt+pVTAGvTEb4wpBn4D7AdKgCpjzLsnricit4jIehFZX1ZW1u/9pseFU3zUvuIHLedXSgUsbxT1xAOXANlAOhApIteduJ4x5jFjzCxjzKzk5OR+7zc9NpyDRxsgdgQ4ozTxK6UCljeKes4B9hljyowxLcArwGme3ml6nJX4jYjVskfb8iulApQ3Ev9+YI6IRIiIAAsBj2fh4fHh1De7qGpogWGT4FAeGOPp3SqllM/xRhn/WuAl4HNgqx3DY57e7/A4q0ln8dEGyJwDDUe0glcpFZC80qrHGPNTY8wEY8xkY8z1xpgmT+8zPS4cgINHG2HkPGtmwSpP71YppXxOQPTchY6JvwHisyAmQxO/UiogBUziT4x04gx2WIlfBLLmQeFqLedXSgWcgEn8IsLwuHCrjB+s4p66Mij/wruBKaXUIAuYxA8wPC6cA5V24s863XrV4h6lVIDpVeIXEYeIxHgqGE/LSopgX1ktxhhIGAXRaVZxj1JKBZCTJn4ReU5EYkQkEsgD8kXkLs+HNvCyk6Kobmylsr7FKucfOQ8KtJxfKRVYenLFP8kYUw1cCizHGmrhek8G5SmjkiIB2Fdea83Imge1h6BijxejUkqpwdWTxB8iIiFYif8Ne5iFIXmJnG0n/r1ldfaMM63X3e97KSKllBp8PUn8jwIFQCTwsYiMBKo9GZSnZMSHE+wQ9pXbiT9xNAzLgfzXvBqXUkoNppMmfmPM740xw40x5xtLIXD2IMQ24IKDHGQmRhxL/AA5l8L+NVDt0UcCKKWUz+hJ5e7tduWuiMjjIvI5sGAQYvOIUUmRxyf+SZdar/lveCUepZQabD0p6vmGXbm7CIjHqth9wKNReVC2nfhdbruaInmcFvcopQJKTxK/2K/nA08bY7Z1mDfkjE+NoanVTUFFx+Key7S4RykVMHqS+DeIyLtYif8dEYkG3J4Ny3MmpkUDsL2kQ/10zqXWa97Lgx+QUkoNsp4k/puBu4FTjDH1gBO4yaNRedCYYVEEO+T4xJ80FjJPg7WPgavVe8EppdQg6EmrHjeQAdwjIr8BTjPGbPF4ZB4SGhzE6OQotpfUHL/gtNugaj9s10pepZR/60mrngeA24F8e/q+iNzv6cA8aVJ6DHnFVdaYPW3GnWeN37PmER3CQSnl13pS1HM+cK4x5gljzBPAecCFng3Ls6ZnxlFa03RsiGYARxDM+Q4Ub4ADa70XnFJKeVhPR+eM6/A+1gNxDKoZmfEAbCisPH5B7jUQHg8f/kKv+pVSfqsnif9/gY0i8qSIPAVsAH7p2bA8a0JqNBHOID4/MfE7I2HBf0PBv2Hri94JTimlPKwnlbvPA3OAV4CXgblYY/cMWcFBDqZnxrH+xMQPMPMmGD4T3vkvaOhkuVJKDXE9KuoxxpQYY96wp0PAkL8cnpkZz/aSamqbTmi+6XDAhb+D+gr44H+8E5xSSnlQXx+9OGR77raZMyoRt4E1eyq+vDBtGsz+Fqx/Qh/NqJTyO31N/EO+5nNWVgKRziBW7iztfIWz/wsSx8DS66G2i3WUUmoICu5qgYi8SecJXoBEj0U0SJzBDk4bk8TKnWUYYxA54SYmLAaufhb+Mh+W3QFXPWM9rlEppYa4LhM/8Js+Lhsyzpk4jPfyD5NXXM2UjE5aqSaPhwX3wHv/DR/9Cs66e/CDVEqpAdZl4jfGfDSYgXjD4pxU7nktjzc2F3ee+AHm3gYHN8LKB2DMOZAxa3CDVEqpAdbXMn6/EBfh5IyxySzbUoLb3UW1hcMBFz0M0Wnwyv/TJp5KqSEvoBM/wMW56ZRUNfLpvk5a97QJi4Ern4SjB+Clb+gInkqpIS3gE/+iSalEhwWz9LMD3a+YeSpc+CDs+RA+/PngBKeUUh7QXeUu0GXrnipgPfCoMaaxtzsVkTjgb8Bke9vfMMas6e12BkK4M4jLpw/n+c8O8LO6ZuIjnV2vPOPrVnn/6ochYzZMHNJj1SmlAlRPrvj3ArXAX+2pGqgBxtmf++Jh4G1jzARgGrC9j9sZEEtOzaS51c0rG4tPvvJ5D0D6dHj121C+2/PBKaXUAOtJ4j/NGHONMeZNe7oO62lc3wVm9HaHIhILnAE8DmCMaTbGHO3tdgbShNQYckfE8fy6/ceP0d+Z4FD42tMQFAL/vB6a67pfXymlfExPEn+UiGS2fbDfR9kfm/uwz2ygDPi7iGwUkb+JSOSJK4nILSKyXkTWl5WV9WE3vXPN7Ex2l9aybt+Rk68cNwK++lco3W517tIhnJVSQ0hPEv+dwCoRWSEiK4F/A/9pJ+un+rDPYKw7hT8bY6YDdVjP9D2OMeYxY8wsY8ys5OTkPuymdy6alk5MWDDPrN3fsy+MOQfO+jFsWWqN6aOUUkPESSt3jTH/EpGxwAR71s4OFboP9WGfRUCRMabtMVcv0UniH2zhziC+OjODZz4tpKxmEsnRoSf/0hl3QdFn8PbdVseutGmeD1Qppfqpp805ZwI5WBWxXxORr/d1h/awzgdEZLw9ayHWs3y97tpTR9LiMvxz/UmadrZxOODSP0NEIvzjEqudv1JK+biePGz9aayxeU4HTrGn/o5b8D3gWRHZAuQCPvHw9jHDopg7KpHn1u7H1VVP3hNFJcONb4GrxerZ29rk2SCVUqqfTlrUg5XkJ5mTNnfpOWPMJvp/8vCI6+aM5LvPfc5HX5SyYEJKz76UONoa1uHlm+H12+Dyx3QkT6WUz+pJUU8ekOrpQHzFopwUkqNDeebTHlbytplyBZx9D2z9pzWSp1JK+aieXPEnAfkisg5oL8cwxlzssai8KCTIwdWnjOCRFbs5cKSeEQkRPf/y/DuhfCes/F9InQITLvBcoEop1Uc9ueL/GXApVjn8bztMfmvJ7EwEeG5dL6/6HQ64+BFIy4WXbob9n3oiPKWU6peTJn5jzEedTYMRnLekx4WzaFIqz3xaSFVDS+++HBIG170M0SnwwjVQscczQSqlVB91mfhFZJX9WiMi1R2mGhGpHrwQveP7C8dS09jK46v29f7LkUnWoxrdrfD0pVDfg97ASik1SLpM/MaY0+3XaGNMTIcp2hgTM3ghesek9Bi+MjmVJ1bto7KuDyNTpE6B616BmkPw7BXQ6PfnSqXUENGjDlwiEiQi6SKS2TZ5OjBfcMe546hvbuWh97/o2wYyZsGVT0HJZnj2SmiqHdgAlVKqD3rSget7wGHgPeAte1rm4bh8wriUaK49dSTPrN3PzkM1fdvIhPPhq49D0Tp4/mpoaRjYIJVSqpd6csV/OzDeGJNjjJliT1M9HZiv+I9zxxEVGsx9b247+ZDNXcm5FC57FApWwQvXau9epZRX9STxH8B64lZAio908p+LxvHJngpe/rwHD2rpytSvwcV/gD0fwD9vgNa+jGitlFL915MOXHuBlSLyFsd34HrQY1H5mGtPHcmbm0u4781tnD4midTYsL5taMb14GqCt+6EV75pFQEFhQxssEopdRI9ueLfj1W+7wSiO0wBw+EQfn3FVFpcbv7r1a19L/IBOOWbsPh/If91eOar0HB0wOJUSqme6Ml4/PcNRiC+Lispkh8unsDPl+Wz9LMDXD27Hw2b5n4HwmLhze/DE4vh2hchLiAaSimlfEB3Hbgesl/fFJE3TpwGLUIfcuNpWZw+Jol7X9/GhsJ+dsqafq3Vzr+6BP66EIo3DEyQSil1EtJVsYWIzDTGbBCRMztbPpjDNsyaNcusX79+sHbXraP1zVzyx9XUNbl447Z5pMeF92+DZTutDl61ZdZzfCdeNDCBKqUCnohsMMZ8aQj87nrubrBfA26snu7ERTh5/IZZNLW4+H//WE99c2v/Npg8Hr75AaRMgqXXwfK7we0emGCVUqoTPenANVZEXhKRfBHZ2zYNRnC+asywaH6/ZDr5JdXc9eKW/lX2AkQNgxuWwYyvw9o/wwtLdHwfpZTH9KRVz9+BPwOtwNnAP4BnPBnUUHD2hGH8+CsTeGtrCf/3zs7+J39nBFz0e/jKr2H3B/DombBnxcAEq5RSHfQk8YcbYz7Aqg8oNMb8DNAnjAD/b/4orj5lBH9auYcHlu/of/IXgVO/Bd94GwRrZM+Vv4KBe+qlUkr1qANXk4g4gF0ichtQDER5NqyhQUS4/7IphAQ5ePTjvYQGO7jj3HFIf5+3mzELvr0a/nUXrLwfDnwKF/4O4rMGJG6lVGDr6Vg9EcD3gZnAdcANngxqKHE4hPsuzuFrszL4/Ye7+dHLW2huHYDK2bAYuOwvcMFv4cA6+NNc+OQRcPWzMlkpFfC6TfwiEgRcZYypNcYUGWNuMsZ81RijzxTswOEQHrh8Kt9bMIZ/ri/ihifWUVXfyyd3dUbE6un73bWQfQa8+xN4/Bw4tLX/21ZKBazuOnAFG2NcwOmDGM+Q5XAIdy4az4Nfm8aGwkou+9NqCsrrBmbjsRmw5AW44u9QVWRV/L73Ux3fXynVJ91d8a+zXzfavXWvF5HL26bBCG4ounxGBs9881Qq65u59E+rWbdvgJplisDky+G76yB3Cax+CB6aDBue0nb/Sqle6UkZfxhQASwALgQusl9VF2ZnJ/Dqd+aREOnk2r99yssbigZu4xEJcMkf4RvvQvJEe7yfRVD4ycDtQynl17pL/MNE5D+APGCr/brNfs0bhNiGtKykSF69dR6nZCVw54ub+cWyfBpbXAO3g8xT4aZ/WSeBqiL4+1fg6cu0/F8pdVLdJf4grGabUVjDMEedMKmTiI0I4alvzObrc0fyt1X7uPiRVeQVD+AzbURg+nXwnTVw9j2wfy385XT4+wVQuGbg9qOU8ivdDdL2uTFmxiDH0ylfGqStr1buLOVHL2+horaZ7y8cy3fOGk1wUI+edd9zDZWw6new6XmoK4Ux58Cos2D69RAeN7D7Ukr5vK4Gaesu8W80xkz3eGQ94A+JH6CqvoV738jj9U0HmZYRy2+unMbYFA8806a5DtY9Bh//FpprwBECZ90Ns2+x+gcopQJCXxJ/gjHGJ0YK85fE3+atLSXc89pW6ppd/Oeicdw0L5uQgb76B2iuh09+D1+8Awc/h/AEmHqV9SAYffCLUn6v14nf0+zOYeuBYmNMt62E/C3xA5TVNPHjV7by/vbDjBkWxX0X5zBvTJJndmaM9aCXlQ/Ang/BuCBuJMy73WoiGh7vmf0qpbzKFxP/fwCzgJhATPwAxhg+2F7Kz5fls/9IPedPSeUnF0xieH8f7tKdqiJYdgfsevfYvOQJcOaPYNKl4PDAnYdSyit8KvGLSAbwFPBL4D8CNfG3aWxx8deP9/LHlbsBuO3sMXxz/ijCQoI8t9PWJnj7btj1HriaofYwRCZbQ0TkXA7J4zy3b6XUoPC1xP8S8L9YzUT/s7PELyK3ALcAZGZmziwsLBzcIL2gqLKeX761neV5h8hMiODeCyexcOKw/o/2eTJuF2x/A9Y/Afs+tualTYN5P4DxX4EQD96BKKU8xmcSv4hcCJxvjPmOiJxFF4m/I3+/4j/Rv3eV8bM3trGnrI5TsxO4c9F4ZmcnDM7Oy76wWgTtWAY1JeCMhtQpMO1qmHIFOCMHJw6lVL/5UuL/X+B6rCd6hQExwCvGmOu6+k6gJX6A5lY3z60t5JEVeyivbWL+2CTuXDSe3BFxgxOA2wUFq2Dri9bU2gjB4TD2XJhwofVQeGfE4MSilOoTn0n8x+1cr/hPqqHZxdOfFvDnlXuorG/hnInDuOPcceSkxw5eEG43FK6C/Ndh+5tWfYAzyhoqOut0yL1GWwYp5YM08Q9xtU2tPLl6H499vJfqxlYumJLGD84Z65kOYN1xu2DncusksPNf0FwLIZEQOxwSx8DCe2HYxMGNSSnVKZ9M/D2lif+YqoYWHv/3Xh5ftY/6FheX5g7n9oVjyUryQtm7qxWKPoPNz8PnTx2bnzUfxiyE8edD0jhrTCGl1KDTxO9njtQ18+jHe3jqkwJaXIYrZmTwvYVjyIj3Urm7qwXKdsAXb8PmF6DCappKTAZMvBBGzIYx5+qQEUoNIk38fqq0ppE/rdjDc2v34zaGi6el860zRzM+dZCLgE5UscdqGpr/GhSsBrf9KMoRc2DC+TB6IaRO9mqISvk7Tfx+rqSqgb9+vI8XPttPfbOLBROG8a0zRjE7O8Hz/QBOpqkWPv0THN5m3RWU7bDmD5tk9RYetwhSp2mvYaUGmCb+AFFZ18zTnxby5CcFHKlrZnpmHN8+czTnTkzB4fCBsnZjoDQf9q60Wgjtt58bEJlsDSM99lwYvUBbCSk1ADTxB5iGZhcvbjjAYx/vpaiygVFJkdxyxigunT7cs0NB9FZVkVUUtPs92P2+9UwBcVjjB41bbD1PYOQ8CArxdqRKDTma+ANUq8vN8rxDPPrxHvKKq0mMdHLdnJFcP3ckSVGh3g7veG6XNYrorvesQeRKNlnzndHWnUD2fBh/AUSneDVMpYYKTfwBzhjDmj0VPL5qHx/sKMUZ7ODS3HRuPn2U9yuCu1JXAfs+su4Etr8JTdWAwPAZVuXwyLkwfCaEDWJnNqWGEE38qt2eslr+vnofL20oorHFzdxRiXxzfjZnjx/mG/UAnTHGqhTe9hrsXWH1HzBucARD9plW/UDmHEifrv0GlLJp4ldfUlnXzAufHeAfawooqWokKzGC6+aM5IqZGcRFOL0dXvfqymH/p3BgrXU3ULnPmh+RZBUJpc+w6gdScsDhQ3UaSg0iTfyqSy12PcDTawr4rKCS0GAHl8/I4MbTsny3GOhE1Qetp4vtWQGFq62RRcE6EYz/inU3MHoBxKQP7H4PrAMJgoyZA7tdpQaAJn7VI9tLqnnqkwJe3VhMU6ubU7LiuebUTL4yOc23WgN1x+2GonVW7+Ed/7IGmGusspbFZVongJzLIXMuBPfzzuZndv3Cz6r6tx2lPEATv+qVI3XNvLThAM+t3U9BRT1xESFcMSODJadmMjo5ytvh9Y7bZV2Zt/UkLt0OGGuY6cw5VtHQqLN7Xz9gDNwXZ73XxK98kCZ+1Sdut2HN3gqeW7ufd7YdotVtmDMqgWtPHcninFScwUOwt21jtdVcdN9HULQBSrdZ82MyrBZDbcNNJ0/o/kRQVw7/N9p6/6NCCI/zeOhK9YYmftVvpTWNvLi+iOfX7aeosoHESCdXzhrBNbMzyUwcwg9lqT5o9R3Y95F1Z1B1wJofmWw9dGb4TBh3HkQlH/+90h3wp1Ot97estO4YlPIhmvjVgHG7DR/vKuPZtfv5YPth3Abmj03i2lMzWTgxhZCgIXgX0MYYa0yhvSutFkO73oPWBmtZdBpMuMDqSZx9JpRthycvsJZNuwYu/gMEBXstdKVOpIlfeURJVQNLPzvA0s8OUFLVyLDoUK46ZQRXnTLCe0NEDyS3Gw5vhd0fWH0Hdn8AriZAAPv/ztjFsOsda9C5hT+FsYt0wDnlEzTxK49qdblZubOMZ9cWsvKLMgDOHj+Ma2ZncvaEYQT5asew3nK1QMlmq+nolqXQ0gC3fWbVGfzrh1BXahURjTrLqjAeffbANyFVqoc08atBc+BIvXUXsP4AZTVNpMWG8bVZI/jqjIyhXRdwIler1Xu4rUloa7P1OModb1lFRXWl1vyk8dZYQ2MWQsYpEDpE+kaoIU8Tvxp0LS437+cf5rl1+1m1uxxjYM6oBK6YOYLFOSlEh/nxiJvtdQUrrLuDglXgaraWJU+0OpWNmG3VFTj96GSofIomfuVVxUcbeG1jMf9cf4DCinrCQ4K4YGoaS2aPYEZmvPcfFuNpzXVQ+IlVYbxnBRz83LpbAKtuYNRZ1rOK06dDTJpXQ1X+QxO/8gnGGD7ff5SXNhzgjU0HqWt2MXZYFFedMoKLc9MZFh3m7RAHR1ONNepo2RfHBp1zt1rLolKsCuJRZ1l3BbEjdOA51Sea+JXPqWtqZdmWgzy/7gCbDhzFITBvTBKXTR/OopxUokIDqGlkQ6V1Etj1rlVEVLjaHoYaiBwGmadadwSZcyBlsg48p3pEE7/yabtLa3ht40Fe21RMUWUDYSEOFk1K5bLpwzl9bNLQ7hvQF24XFK2HgxutoSYOfAr1FdYyZ7TVwzhjllVZPHzWlzuXKYUmfjVEGGPYUFjJqxuLeWtrCUfrW0iMdHLh1DQunT6c3BFx/l8f0JWjB6xnFO9fY50UDm8D47KWxY2E9FzrriAtF5LG6hASShO/GnqaW9189EUZr20s5r3th2ludZOVGMElucO5dPpwspMivR2idzXXW30Kij6D4vXWyaC62FrmCIakcfadwWyrriBpvHYsCzCa+NWQVt3Ywtt5h3htYzFr9lZgDOSOiOPS3HTOn5oWOJXC3TEGSvOhZAsczoPyL6yTQkOltTw0FlKnQEIWDMuBtGkw4lQdZsKPaeJXfqOkqoE3Nx/k1Y0H2V5iVYDOzkrgkunpnJeTSqKvPUTem4yBij3W8wkOrLOGpC7feexk4AiBxNHWHUHGbEgYBWlTtZOZn9DEr/zSzkM1vJ13iNc3F7O3rI4ghzBnVAJfmZzGeZNTSdKTwJe5XVBbarUcOrTV6lxWtgOaa4+tEzfSelDNsAmQeZp1p6AdzYYcTfzKrxljyC+pZvnWQ/xrawl7y+vam4deMCWNBROHaXFQd1qboKrIujso2Wy1JireALWH7BXEenpZ8gTrZJA8wepspvUGPs1nEr+IjAD+AaRgDW/4mDHm4e6+o4lf9YYxhp2Ha1i2uYTXNxdz4EgDQQ5h7qhEFk9OZfGkFIbF6EmgR6qKrYrjsp1WMVHZTqjYdWz4CUcIJI6xHmqfPMEqNkoaa80LCfdu7MqnEn8akGaM+VxEooENwKXGmPyuvqOJX/WVMYa84mqW55Xwdt4h9pbXIQIzM+M5b3Iq505KYWRigLcO6i1Xq1VxfGAtVBZYJ4PDecceYAOAQPxI62SQNM56TR5vvQ+L8VbkAcdnEv+XAhB5HXjEGPNeV+to4lcDwRjDrtJa3s47xPK8Q+0Vw+NSojh3UgoLJqQwNSM28DqLDZTmejiyB8p3WSeGsh1Wb+SOdwgA0enWSSAuE2KGW3cJablWxbIWGw0on0z8IpIFfAxMNsZUd7WeJn7lCQeO1PNe/mHeyz/MuoIjuNyGpKhQ5o9NYnFOCvPGJPn3CKKDxdUKRwvtE8FOayq3X1vqj60XEmGdBMLiICEbEkZbnxNGWZXNoVFeO4ShyucSv4hEAR8BvzTGvNLJ8luAWwAyMzNnFhYWDnKEKpBU1Daxanc57+YfZtWucqoaWgh2CKeOSmBxTirzxyZrh7GBZgzUH7E6nZVstvoglH8BjdVwZC/Ulx+/fmSydQKIz7KKkeKzjn2OGa79ETrhU4lfREKAZcA7xpgHT7a+XvGrwdTicrOhsJKVO8t4O6+EggrrqnR8SjSnj03i9DFJzM5OIDKQBpHzhsYqq5VR5T6oLLTqE462vR44NlwFWD2VYzOOPxnEZUKQ07p7iBsZkHULPpP4xRpo5SngiDHmBz35jiZ+5S3GGPbbRUIf7ihlfWElza1uQoKEhRNSmD8uiXmjkxiZGBG4Ywh5g6vVulPoeDJoOzlUFnz5bgGsoqToNOsEEZlkvUalQni8dZKISICIRGvyk9FPfSnxnw78G9gK2E+i4L+MMf/q6jua+JWvaGxxsaGwkvfyD/N23iEOVTcCMDwunNNGJzJvTBKnjUnUPgPe1lRr9UtobbSKjY7utzqtVRdbU1Ux1Bzs/LsSBNGp1gkgapjVizku0z5JxFnzo1MhNMaeoiDYNzsK+kzi7wtN/MoXGWPYW17HJ7vLWb27gjV7K6hqaAFg7LAo5o1J4oxxSczOTgysZwsMFa1N1pPRGiqtu4TGo1BXYXVaqzlknSjqSq3lR/d3v62wWKtPQ2i0PcVYr85Iqz9D+xRxwusJ8wCCQo69rz9idZQL6duFhCZ+pTzM5TbkH6xm9Z5yVu8u57OCIzS2uAlyCJOHxzJ3VCKnZMUzZXisdiAbappqrSapjUehrtw6KTTVWA/LaXvvbrHWa6qxpypoabCneuu1tbH3+/7uZ5A8rk9ha+JXapC1FQut2VPBp3sr2Fx0lBaX9f9tWkYsp41JYnJ6LDNHxpMSE6p1BIHA7bKSf8eTQdtrs9201dV8rJlreByMmNPnpqxdJX69/1TKQ8JCgpg3Jol5Y5IAqG9uJa+4ms8KjvD+9sP89eO9tLqtE8HwuHBOzU5g+sh4po+IY3xqtHYk80eOIKv4x+ndpsF6xa+UlzS1uthRUsPn+yv5dG8FGworKa+1eriGBjuYnhnHKVkJTEqLYWZWvFYYq17Toh6lfJwxhqLKBjYXHeXzwqOsK6hge0kNLvuuIC02jKkZsUzNiGNaRhxTMmKJDdeexaprWtSjlI8TEUYkRDAiIYILp6YDVj1Bfkk1nxdWsqWoii1FR3ln2+H272QlRjA1I46pGbFMGxHHpLQY7VimTkr/hSjlw8JCgpiRGc+MzPj2eVX1LWwpPtp+Ivis4AhvbLbapItAdlIkOemx5KTHMDk9lknpMSREOr11CMoHaeJXaoiJjQhh/thk5o9Nbp9XWtPIlgNVbDtYTd7BKj4vrOTNzcc6KA2LDmVCWgyT0mKYlG69ZidFEuTQlkSBSBO/Un5gWHQY50wK45xJKe3zKuuayS+pZntJNdtLatheUs3je/a2NykNDXaQnRTJ+NRoctJjmJAaw7iUaG1aGgA08Svlp+Ijncc1JwVobnWzu7SWbQer+OJwDbtLa1m37wivbzp2dxAdFszo5CjGDItidHIUo5MjyUyMICM+Qnsg+wn9FZUKIM5gh1XUk378SJVH6prZeaiGXaU17Dpcy56yWv69q4yXNhQdt15SlJORiZGMTIhgZGIkWUkRZCZEkJUYSVxEiN4pDBGa+JVSJEQ6mTs6kbmjE4+bX9PYwt6yOg5U1nPgSAP7j9RRUF7Pp3sreHVTMR1bg0eHBZOVGMnIxAh7imREfAQZ8eGkxoZphzQfoolfKdWl6LAQpo2IY9qIuC8ta2xxUVRZT0F5PQUVdew/Uk9BRT1bi6tYnneovf8BWK2NUqLDSI8LIz0unOHx4WTEhZMRH0FaXBhpseHEhAXrHcMg0cSvlOqTsJAgxgyLZsyw6C8ta3G5Ka5soKiygeKj9RRXNnCwqpHiyga2Flfx7rbDNLvcx30nwhlEWqx1YkiLDWNYdBhpcWGkRIeREOUkOSqUlJgwnMF659BfmviVUgMuJMhBVlIkWV08rtLtNpTVNlFUWc/Bo40cqmrkYFUDJUcbKaluZOehMirqmo+7a4Bjdw5J0U6SokJJiLRek6KcJEaGkmi/JkQ5SYx0EhbiHw9UGWia+JVSg87hEFJiwkiJCWPmyM7XcbsNh2saKatpoqK2mcPVje13DWW11rxdh2spq22iudXd6TYinEGkxoYRGx5CXHgIcRFOYsNDiI9wEhcRYk9O4sJDiA0PITosmOiwEL+/q9DEr5TySQ6HkBYbTlpseLfrGWOobWqloraZijrrhHCkrpmKOuu1uLKBuuZWymub2VVaS1V9CzVNrd1uMzTYQXRY24nAmqJCg9vnRYUGExYSRHhIEOHOY68RziDCQoIIDXYQ7HDgDBbCncGEBTsIDQki2CGEBDm83nFOE79SakgTETshh3RZtHSiFpebqoYWjtY3c7S+hcr6FqobWqhpbKG6sZXaplZqGluoaWy1pxbKapraP9c1t9Kf8S1FINghBDscBAeJ9T7IQYhDCAoSQhwORKCh2cXSb81lREJE33fWCU38SqmAExLksOsG+vasXGMMTa1uGltcNLS4aGh2Ud9svW9scdHc6qbFZWh2uWlobm1ft8VlaHUZXG43LW5Dq8tNq9ua1+p226+GFpcbA4QFBxEcNPB3B5r4lVKql0SEsBCrWCfO28H0gX/XYCillPoSTfxKKRVgNPErpVSA0cSvlFIBRhO/UkoFGE38SikVYDTxK6VUgNHEr5RSAUZMf/odDxIRKQMK+/j1JKB8AMMZCvSYA0OgHXOgHS/0/5hHGmOST5w5JBJ/f4jIemPMLG/HMZj0mANDoB1zoB0veO6YtahHKaUCjCZ+pZQKMIGQ+B/zdgBeoMccGALtmAPteMFDx+z3ZfxKKaWOFwhX/EoppTrQxK+UUgHGbxO/iJwnIjtFZLeI3O3teAaKiIwQkRUiki8i20Tkdnt+goi8JyK77Nd4e76IyO/tv8MWEZnh3SPoOxEJEpGNIrLM/pwtImvtY1sqIk57fqj9ebe9PMurgfeRiMSJyEsiskNEtovIXH//nUXkDvvfdZ6IPC8iYf72O4vIEyJSKiJ5Heb1+ncVkRvs9XeJyA29icEvE7+IBAF/BL4CTAKWiMgk70Y1YFqBO40xk4A5wHftY7sb+MAYMxb4wP4M1t9grD3dAvx58EMeMLcD2zt8/hXwO2PMGKASuNmefzNQac//nb3eUPQw8LYxZgIwDevY/fZ3FpHhwPeBWcaYyUAQcDX+9zs/CZx3wrxe/a4ikgD8FDgVmA38tO1k0SPGGL+bgLnAOx0+/xj4sbfj8tCxvg6cC+wE0ux5acBO+/2jwJIO67evN5QmIMP+D7EAWAYIVo/G4BN/c+AdYK79PtheT7x9DL083lhg34lx+/PvDAwHDgAJ9u+2DFjsj78zkAXk9fV3BZYAj3aYf9x6J5v88oqfY/+A2hTZ8/yKfWs7HVgLpBhjSuxFh4AU+72//C0eAn4IuO3PicBRY0yr/bnjcbUfs728yl5/KMkGyoC/28VbfxORSPz4dzbGFAO/AfYDJVi/2wb8+3du09vftV+/t78mfr8nIlHAy8APjDHVHZcZ6xLAb9rpisiFQKkxZoO3YxlEwcAM4M/GmOlAHcdu/wG//J3jgUuwTnrpQCRfLhLxe4Pxu/pr4i8GRnT4nGHP8wsiEoKV9J81xrxizz4sImn28jSg1J7vD3+LecDFIlIAvIBV3PMwECciwfY6HY+r/Zjt5bFAxWAGPACKgCJjzFr780tYJwJ//p3PAfYZY8qMMS3AK1i/vT//zm16+7v26/f218T/GTDWbg3gxKogesPLMQ0IERHgcWC7MebBDoveANpq9m/AKvtvm/91u3XAHKCqwy3lkGCM+bExJsMYk4X1W35ojLkWWAFcYa924jG3/S2usNcfUlfGxphDwAERGW/PWgjk48e/M1YRzxwRibD/nbcds9/+zh309nd9B1gkIvH2ndIie17PeLuSw4OVJ+cDXwB7gJ94O54BPK7TsW4DtwCb7Ol8rLLND4BdwPtAgr2+YLVw2gNsxWox4fXj6MfxnwUss9+PAtYBu4EXgVB7fpj9ebe9fJS34+7jseYC6+3f+jUg3t9/Z+A+YAeQBzwNhPrb7ww8j1WH0YJ1Z3dzX35X4Bv2se8GbupNDDpkg1JKBRh/LepRSinVBU38SikVYDTxK6VUgNHEr5RSAUYTv1JKBRhN/EoBIuISkU0dpgEb0VVEsjqOxKiUtwWffBWlAkKDMSbX20EoNRj0il+pbohIgYj8WkS2isg6ERljz88SkQ/tMdI/EJFMe36KiLwqIpvt6TR7U0Ei8ld7rPl3RSTcawelAp4mfqUs4ScU9VzVYVmVMWYK8AjWKKEAfwCeMsZMBZ4Ffm/P/z3wkTFmGtbYOtvs+WOBPxpjcoCjwFc9ejRKdUN77ioFiEitMSaqk/kFwAJjzF57cLxDxphEESnHGj+9xZ5fYoxJEpEyIMMY09RhG1nAe8Z6yAYi8iMgxBjzi0E4NKW+RK/4lTo508X73mjq8N6F1q8pL9LEr9TJXdXhdY39/hOskUIBrgX+bb//ALgV2p8RHDtYQSrVU3rVoZQlXEQ2dfj8tjGmrUlnvIhswbpqX2LP+x7W07HuwnpS1k32/NuBx0TkZqwr+1uxRmJUymdoGb9S3bDL+GcZY8q9HYtSA0WLepRSKsDoFb9SSgUYveJXSqkAo4lfKaUCjCZ+pZQKMJr4lVIqwGjiV0qpAPP/AaHmlzwFHdM8AAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "vanilla_loss = [i[\"loss\"] for i in n[\"train\"]]\n", "vanilla_epochs = [i[\"epoch\"] for i in n[\"train\"]]\n", @@ -979,7 +598,7 @@ }, { "cell_type": "markdown", - "id": "1842f1f9", + "id": "eb286e47", "metadata": {}, "source": [ "### Training Stats\n", @@ -990,74 +609,10 @@ }, { "cell_type": "code", - "execution_count": 30, - "id": "9a196443", + "execution_count": null, + "id": "966b058c", "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
train_runtimetrain_samples_per_secondtrain_steps_per_secondtrain_lossepoch
PT + SM DDP1305.67741216.2272.2982.3402011000.0
PT + Compiler852.87781861.9321.1733.2869791000.0
\n", - "
" - ], - "text/plain": [ - " train_runtime train_samples_per_second \\\n", - "PT + SM DDP 1305.6774 1216.227 \n", - "PT + Compiler 852.8778 1861.932 \n", - "\n", - " train_steps_per_second train_loss epoch \n", - "PT + SM DDP 2.298 2.340201 1000.0 \n", - "PT + Compiler 1.173 3.286979 1000.0 " - ] - }, - "execution_count": 30, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "import pandas as pd\n", "\n", @@ -1066,18 +621,10 @@ }, { "cell_type": "code", - "execution_count": 31, - "id": "34982525", + "execution_count": null, + "id": "8594ba1f", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SageMaker Training Compiler is about 34% faster in terms of total training time.\n" - ] - } - ], + "outputs": [], "source": [ "speedup = (\n", " (n[\"summary\"][\"train_runtime\"] - o[\"summary\"][\"train_runtime\"])\n", @@ -1091,7 +638,7 @@ }, { "cell_type": "markdown", - "id": "8ec367e0", + "id": "8c529a52", "metadata": {}, "source": [ "### Total Billable Time\n", @@ -1101,8 +648,8 @@ }, { "cell_type": "code", - "execution_count": 32, - "id": "fa66ac61", + "execution_count": null, + "id": "ae266273", "metadata": {}, "outputs": [], "source": [ @@ -1116,55 +663,10 @@ }, { "cell_type": "code", - "execution_count": 33, - "id": "66b1c0b1", + "execution_count": null, + "id": "b963fb0f", "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PT + SM DDPPT + Compiler
BillableSecs25821980
\n", - "
" - ], - "text/plain": [ - " PT + SM DDP PT + Compiler\n", - "BillableSecs 2582 1980" - ] - }, - "execution_count": 33, - "metadata": {}, - "output_type": "execute_result" - } - ], + "outputs": [], "source": [ "Billable = {}\n", "Billable[\"PT + SM DDP\"] = BillableTimeInSeconds(native_estimator.latest_training_job.name)\n", @@ -1174,18 +676,10 @@ }, { "cell_type": "code", - "execution_count": 34, - "id": "7e0373db", + "execution_count": null, + "id": "e8ff8331", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SageMaker Training Compiler integrated PyTorch was 23% faster in summary.\n" - ] - } - ], + "outputs": [], "source": [ "speedup = (Billable[\"PT + SM DDP\"] - Billable[\"PT + Compiler\"]) * 100 / Billable[\"PT + SM DDP\"]\n", "print(f\"SageMaker Training Compiler integrated PyTorch was {int(speedup)}% faster in summary.\")" @@ -1193,7 +687,7 @@ }, { "cell_type": "markdown", - "id": "ef17c2ac", + "id": "27b32da2", "metadata": {}, "source": [ "## Clean up\n", @@ -1203,8 +697,8 @@ }, { "cell_type": "code", - "execution_count": 35, - "id": "b96c6215", + "execution_count": null, + "id": "dd53b176", "metadata": {}, "outputs": [], "source": [ @@ -1225,7 +719,7 @@ }, { "cell_type": "markdown", - "id": "26636a53", + "id": "e3e6b15d", "metadata": {}, "source": [ "Also, to find instructions on cleaning up resources, see [Clean Up](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html) in the *Amazon SageMaker Developer Guide*." From ab7e74c26ce182f089b2b349cdcc2fbfa5357a71 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Tue, 18 Oct 2022 20:15:26 +0000 Subject: [PATCH 6/9] change dataset from sst2 to wikitext --- ...nguage-modeling-multi-gpu-multi-node.ipynb | 121 +++++++++--------- .../scripts/run_clm.py | 2 + .../scripts/run_mlm.py | 3 + 3 files changed, 66 insertions(+), 60 deletions(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index 63a9eec288..3fa95f1e6d 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -2,15 +2,15 @@ "cells": [ { "cell_type": "markdown", - "id": "930e721f", + "id": "922da91e", "metadata": {}, "source": [ - "# Compile and Train the GPT2 Model using the Transformers Trainer API with the SST2 Dataset for Multi-Node Multi-GPU Training" + "# Compile and Train the GPT2 Model using the Transformers Trainer API with the wikitext Dataset for Multi-Node Multi-GPU Training" ] }, { "cell_type": "markdown", - "id": "56173568", + "id": "e2762d09", "metadata": {}, "source": [ "1. [Introduction](#Introduction) \n", @@ -25,7 +25,7 @@ }, { "cell_type": "markdown", - "id": "a7a7be64", + "id": "2d6f733f", "metadata": {}, "source": [ "## SageMaker Training Compiler Overview\n", @@ -38,7 +38,7 @@ "\n", "## Introduction\n", "\n", - "In this demo, you'll use Hugging Face's transformers and datasets libraries with Amazon SageMaker Training Compiler to train the gpt-2 model on the Stanford Sentiment Treebank v2 (SST2) dataset. To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on. \n", + "In this demo, you'll use Hugging Face's transformers and datasets libraries with Amazon SageMaker Training Compiler to train the gpt-2 model on the wikitext dataset. To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on. \n", "\n", "**NOTE:** You can run this demo in SageMaker Studio, SageMaker notebook instances, or your local machine with AWS CLI set up. If using SageMaker Studio or SageMaker notebook instances, make sure you choose one of the PyTorch-based kernels, Python 3 (PyTorch x.y Python 3.x CPU Optimized) or conda_pytorch_p36 respectively.\n", "\n", @@ -47,7 +47,7 @@ }, { "cell_type": "markdown", - "id": "41df9c79", + "id": "6fe11605", "metadata": {}, "source": [ "## Development Environment " @@ -55,7 +55,7 @@ }, { "cell_type": "markdown", - "id": "6e1e5d50", + "id": "293eff00", "metadata": {}, "source": [ "### Installation" @@ -63,7 +63,7 @@ }, { "cell_type": "markdown", - "id": "66ee6d74", + "id": "86e7a516", "metadata": {}, "source": [ "This example notebook requires the **SageMaker Python SDK v2.108.0** and **transformers v4.21**." @@ -72,7 +72,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6e4966b4", + "id": "9782c8b3", "metadata": {}, "outputs": [], "source": [ @@ -82,7 +82,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3469acdb", + "id": "de3ca816", "metadata": {}, "outputs": [], "source": [ @@ -92,7 +92,7 @@ { "cell_type": "code", "execution_count": null, - "id": "39b17a6f", + "id": "ad210e7d", "metadata": {}, "outputs": [], "source": [ @@ -108,7 +108,7 @@ }, { "cell_type": "markdown", - "id": "cc38670d", + "id": "64570f5d", "metadata": {}, "source": [ "**NOTE:** Copy and run the following code if you need to upgrade ipywidgets for datasets library and restart the kernel. This is needed if the installation is not applied to the current kernel.\n", @@ -124,7 +124,7 @@ }, { "cell_type": "markdown", - "id": "68064ca2", + "id": "c47f3664", "metadata": {}, "source": [ "### SageMaker Environment " @@ -133,7 +133,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f841807d", + "id": "88e13af0", "metadata": {}, "outputs": [], "source": [ @@ -158,7 +158,7 @@ }, { "cell_type": "markdown", - "id": "c0d249c6", + "id": "2025db42", "metadata": {}, "source": [ "## SageMaker Training Job\n", @@ -171,13 +171,13 @@ { "cell_type": "code", "execution_count": null, - "id": "32ef6ab0", + "id": "b07a2709", "metadata": {}, "outputs": [], "source": [ "# Here we configure the training job. Please configure the appropriate options below:\n", "\n", - "EPOCHS = 800\n", + "EPOCHS = 10\n", "\n", "# Choose between Causal Language Model and Masked Language Model\n", "LANGUAGE_MODELING_LOSS = \"clm\" # or \"mlm\"\n", @@ -199,7 +199,7 @@ }, { "cell_type": "markdown", - "id": "a7935fae", + "id": "46c5650c", "metadata": {}, "source": [ "First, we define some basic parameters common to all estimators.\n", @@ -210,7 +210,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d3402d03", + "id": "63846356", "metadata": {}, "outputs": [], "source": [ @@ -220,8 +220,8 @@ " instance_type=INSTANCE_TYPE,\n", " instance_count=NUM_INSTANCES,\n", " role=role,\n", - " volume_size=100,\n", " py_version=\"py38\",\n", + " volume_size=512,\n", " disable_profiler=True, # Disabling SageMaker Profiler to avoid overheads during benchmarking\n", " debugger_hook_config=False, # Disabling SageMaker Debugger to avoid overheads during benchmarking\n", " base_job_name=\"trcomp-pt-example\",\n", @@ -242,7 +242,7 @@ }, { "cell_type": "markdown", - "id": "eded57be", + "id": "359ea82b", "metadata": {}, "source": [ "Next, we define some basic arguments to be passed to the training script." @@ -251,7 +251,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4430ec63", + "id": "c45ecc3b", "metadata": {}, "outputs": [], "source": [ @@ -259,24 +259,25 @@ "hyperparameters = {\n", " MODEL_CONFIG: MODEL_NAME,\n", " \"tokenizer_name\": TOKENIZER_NAME,\n", - " \"dataset_name\": \"glue\",\n", - " \"dataset_config_name\": \"sst2\",\n", + " \"dataset_name\": \"wikitext\",\n", + " \"dataset_config_name\": \"wikitext-103-v1\",\n", " \"do_train\": True,\n", " \"do_eval\": False,\n", " \"num_train_epochs\": EPOCHS,\n", + " SEQ_LEN_ARG: 512,\n", " \"overwrite_output_dir\": True,\n", + " \"save_strategy\": \"no\",\n", " \"evaluation_strategy\": \"no\",\n", " \"logging_strategy\": \"epoch\",\n", " \"output_dir\": \"/opt/ml/model\",\n", " \"dataloader_drop_last\": True,\n", - " SEQ_LEN_ARG: 512,\n", " \"preprocessing_num_workers\": 12,\n", "}" ] }, { "cell_type": "markdown", - "id": "63a0ad37", + "id": "f15ae3eb", "metadata": {}, "source": [ "In the following sections, we will create estimators and start training." @@ -284,7 +285,7 @@ }, { "cell_type": "markdown", - "id": "ce809ad9", + "id": "97d35764", "metadata": {}, "source": [ "### Training with Native PyTorch + SM DDP" @@ -292,7 +293,7 @@ }, { "cell_type": "markdown", - "id": "8fa6201d", + "id": "4b3575d9", "metadata": {}, "source": [ "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training.\n", @@ -303,13 +304,13 @@ { "cell_type": "code", "execution_count": null, - "id": "ff8e4608", + "id": "4778f414", "metadata": {}, "outputs": [], "source": [ "from sagemaker.pytorch import PyTorch\n", "\n", - "hyperparameters[\"per_device_train_batch_size\"] = 16\n", + "hyperparameters[\"per_device_train_batch_size\"] = 13\n", "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", @@ -333,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "adc1af25", + "id": "3a6d8e80", "metadata": {}, "source": [ "### Training with SageMaker Training Compiler " @@ -341,7 +342,7 @@ }, { "cell_type": "markdown", - "id": "3a4e5ef6", + "id": "d288f21d", "metadata": {}, "source": [ "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately." @@ -350,14 +351,14 @@ { "cell_type": "code", "execution_count": null, - "id": "be0e51d0", + "id": "a550ef7b", "metadata": {}, "outputs": [], "source": [ "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", "\n", "# with SageMaker Training Compiler we are able to fit a larger batch into memory\n", - "hyperparameters[\"per_device_train_batch_size\"] = 32\n", + "hyperparameters[\"per_device_train_batch_size\"] = 25\n", "\n", "# The original LR was set for a batch of 32. Here we are scaling learning rate with batch size.\n", "hyperparameters[\"learning_rate\"] = (\n", @@ -367,11 +368,11 @@ "# configure the training job\n", "optimized_estimator = HuggingFace(\n", " compiler_config=TrainingCompilerConfig(),\n", - " **estimator_args,\n", " transformers_version=\"4.21\",\n", " pytorch_version=\"1.11\",\n", " hyperparameters=hyperparameters,\n", " distribution={\"pytorchxla\": {\"enabled\": True}},\n", + " **estimator_args,\n", ")\n", "\n", "# start the training job\n", @@ -381,7 +382,7 @@ }, { "cell_type": "markdown", - "id": "46b6f94f", + "id": "cd07fd0e", "metadata": {}, "source": [ "### Wait for training jobs to complete\n" @@ -390,7 +391,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a9f1e382", + "id": "aa99c58d", "metadata": {}, "outputs": [], "source": [ @@ -406,7 +407,7 @@ }, { "cell_type": "markdown", - "id": "12e0aa73", + "id": "13a6ed8f", "metadata": {}, "source": [ "## Analysis" @@ -414,7 +415,7 @@ }, { "cell_type": "markdown", - "id": "20c2a1eb", + "id": "8f3a568c", "metadata": {}, "source": [ "**Note:** If the estimator object is no longer available due to a kernel break or refresh, you need to directly use the training job name and manually attach the training job to a new HuggingFace estimator. For example:\n", @@ -426,7 +427,7 @@ }, { "cell_type": "markdown", - "id": "cb489539", + "id": "8c238cf7", "metadata": {}, "source": [ "### Load logs of the training job *with* SageMaker Training Compiler" @@ -435,7 +436,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0eac1589", + "id": "ea71078b", "metadata": {}, "outputs": [], "source": [ @@ -447,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "b103b5ce", + "id": "948dba2b", "metadata": {}, "source": [ "### Load logs of the training job *without* SageMaker Training Compiler" @@ -456,7 +457,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8e07e595", + "id": "5d19e18c", "metadata": {}, "outputs": [], "source": [ @@ -468,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "5a64aa04", + "id": "4e186c13", "metadata": {}, "source": [ "### Create helper functions for analysis" @@ -477,7 +478,7 @@ { "cell_type": "code", "execution_count": null, - "id": "12b72ee5", + "id": "aa34ebfd", "metadata": {}, "outputs": [], "source": [ @@ -518,7 +519,7 @@ }, { "cell_type": "markdown", - "id": "7db33694", + "id": "b6862bdc", "metadata": {}, "source": [ "### Plot Optimized vs Native Training Throughput\n", @@ -529,7 +530,7 @@ { "cell_type": "code", "execution_count": null, - "id": "85dcaaf7", + "id": "6e30398b", "metadata": { "scrolled": true }, @@ -550,7 +551,7 @@ { "cell_type": "code", "execution_count": null, - "id": "aa270a8c", + "id": "0a17264c", "metadata": {}, "outputs": [], "source": [ @@ -568,7 +569,7 @@ }, { "cell_type": "markdown", - "id": "db11ce58", + "id": "6796db9d", "metadata": {}, "source": [ "### Convergence of Training Loss\n", @@ -579,7 +580,7 @@ { "cell_type": "code", "execution_count": null, - "id": "5bc8a723", + "id": "d99dc081", "metadata": {}, "outputs": [], "source": [ @@ -598,7 +599,7 @@ }, { "cell_type": "markdown", - "id": "eb286e47", + "id": "fe3cb58a", "metadata": {}, "source": [ "### Training Stats\n", @@ -610,7 +611,7 @@ { "cell_type": "code", "execution_count": null, - "id": "966b058c", + "id": "7bb74930", "metadata": {}, "outputs": [], "source": [ @@ -622,7 +623,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8594ba1f", + "id": "e605617c", "metadata": {}, "outputs": [], "source": [ @@ -638,7 +639,7 @@ }, { "cell_type": "markdown", - "id": "8c529a52", + "id": "973ddcb9", "metadata": {}, "source": [ "### Total Billable Time\n", @@ -649,7 +650,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ae266273", + "id": "0f31fc1a", "metadata": {}, "outputs": [], "source": [ @@ -664,7 +665,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b963fb0f", + "id": "ec70b709", "metadata": {}, "outputs": [], "source": [ @@ -677,7 +678,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e8ff8331", + "id": "0f1bb1ee", "metadata": {}, "outputs": [], "source": [ @@ -687,7 +688,7 @@ }, { "cell_type": "markdown", - "id": "27b32da2", + "id": "3e64821c", "metadata": {}, "source": [ "## Clean up\n", @@ -698,7 +699,7 @@ { "cell_type": "code", "execution_count": null, - "id": "dd53b176", + "id": "0ede2e27", "metadata": {}, "outputs": [], "source": [ @@ -719,7 +720,7 @@ }, { "cell_type": "markdown", - "id": "e3e6b15d", + "id": "8000452a", "metadata": {}, "source": [ "Also, to find instructions on cleaning up resources, see [Clean Up](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html) in the *Amazon SageMaker Developer Guide*." diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm.py index d2a473f6b6..b6591ce4d0 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm.py +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_clm.py @@ -435,6 +435,7 @@ def tokenize_function(examples): remove_columns=column_names, load_from_cache_file=not data_args.overwrite_cache, desc="Running tokenizer on dataset", + keep_in_memory=True, ) if data_args.block_size is None: @@ -484,6 +485,7 @@ def group_texts(examples): num_proc=data_args.preprocessing_num_workers, load_from_cache_file=not data_args.overwrite_cache, desc=f"Grouping texts in chunks of {block_size}", + keep_in_memory=True, ) if training_args.do_train: diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm.py b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm.py index a0d3302566..fd8c9cd978 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm.py +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/scripts/run_mlm.py @@ -454,6 +454,7 @@ def tokenize_function(examples): remove_columns=[text_column_name], load_from_cache_file=not data_args.overwrite_cache, desc="Running tokenizer on dataset line_by_line", + keep_in_memory=True, ) else: # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts. @@ -470,6 +471,7 @@ def tokenize_function(examples): remove_columns=column_names, load_from_cache_file=not data_args.overwrite_cache, desc="Running tokenizer on every text in dataset", + keep_in_memory=True, ) # Main data processing function that will concatenate all texts from our dataset and generate chunks of @@ -503,6 +505,7 @@ def group_texts(examples): num_proc=data_args.preprocessing_num_workers, load_from_cache_file=not data_args.overwrite_cache, desc=f"Grouping texts in chunks of {max_seq_length}", + keep_in_memory=True, ) if training_args.do_train: From 275a0d67c5c804f95d41467d61cf86b809e3ce7a Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Wed, 19 Oct 2022 16:04:03 +0000 Subject: [PATCH 7/9] edit package install and add comments for ptxla --- .../language-modeling-multi-gpu-multi-node.ipynb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index 3fa95f1e6d..d0db4c3174 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -76,7 +76,7 @@ "metadata": {}, "outputs": [], "source": [ - "!pip install \"sagemaker>=2.108.0\" botocore boto3 awscli s3fs typing-extensions \"torch==1.11.0\" pandas numpy --upgrade" + "!pip install \"sagemaker>=2.108.0\" botocore boto3 awscli typing-extensions pandas numpy --upgrade" ] }, { @@ -86,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "!pip install \"transformers==4.21\" datasets --upgrade" + "!pip install \"transformers==4.21\" --upgrade" ] }, { @@ -345,7 +345,10 @@ "id": "d288f21d", "metadata": {}, "source": [ - "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately." + "Compilation through Training Compiler changes the memory footprint of the model. Most commonly, this manifests as a reduction in memory utilization and a consequent increase in the largest batch size that can fit on the GPU. Note that if you want to change the batch size, you must adjust the learning rate appropriately.\n", + "Below, we have scaled the learning rate linearly with the increase in batch size.\n", + "\n", + "**Note**: We are using distribution mechanism pytorchxla which is a compiler aware method of distributed training." ] }, { From 1414bf6639c3eafeba8fd4e287a52b66e54ac6c8 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Wed, 19 Oct 2022 18:35:05 +0000 Subject: [PATCH 8/9] fix comments --- .../language-modeling-multi-gpu-multi-node.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index d0db4c3174..ef9b652861 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -296,7 +296,7 @@ "id": "4b3575d9", "metadata": {}, "source": [ - "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory. We also use Automatic Mixed Precision for faster training.\n", + "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory.\n", "\n", "This example uses HuggingFace training script run_clm.py, which you can find it inside the scripts folder." ] From f67ab79c6e1479c5f64b38c3a00e68c6b0d69463 Mon Sep 17 00:00:00 2001 From: "BruceZhang@eitug" Date: Wed, 19 Oct 2022 18:46:55 +0000 Subject: [PATCH 9/9] fix grammar --- .../language-modeling-multi-gpu-multi-node.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb index ef9b652861..a0f2876580 100644 --- a/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb +++ b/sagemaker-training-compiler/huggingface/pytorch_multiple_gpu_multiple_node/language-modeling-multi-gpu-multi-node.ipynb @@ -296,7 +296,7 @@ "id": "4b3575d9", "metadata": {}, "source": [ - "The batch size below is the maximum batch we could fit into the memory of an ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory.\n", + "The batch size below is the maximum batch we could fit into the memory of a ml.p4d.24xlarge instance. If you change the model, instance type, sequence length, and other parameters, you need to do some experiments to find the largest batch size that will fit into GPU memory.\n", "\n", "This example uses HuggingFace training script run_clm.py, which you can find it inside the scripts folder." ]