SageMaker DP Support

[pacman100]

What does this PR do?

1. Fixes bugs in the current SageMaker support to get it working properly using the latest HF DLC
2. Adds DATA_PARALLEL support. Run accelerate config and answer the questions and choose DATA_PARALLEL for SageMaker Distribution Type. Sample config below with XXXXX values being AWS account specific.

base_job_name: accelerate-sagemaker-1
compute_environment: AMAZON_SAGEMAKER
distributed_type: DATA_PARALLEL
ec2_instance_type: ml.p3.16xlarge
iam_role_name: XXXXX
mixed_precision: fp16
num_machines: 1
profile: XXXXX
py_version: py38
pytorch_version: 1.10.2
region: us-east-1
transformers_version: 4.17.0
use_cpu: false

After above config, just run the below command to run the official NLP example:

cd acceelerate/examples
accelerate launch complete_nlp_example.py

The output logs:

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Looking good at first glance! Let's focus on DataParallel only as AWS told us they were completely changing the API for model parallelism.

[sgugger]

Thanks for working on this! Let's also see what @philschmid thinks since he worked on the sagemaker command in Accelerate.

[plamb-viso]

I will test this with my use case as soon as it gets merged into main

[philschmid]

LGTM! How should we maintain the SAGEMAKER_PYTHON_VERSION, SAGEMAKER_PYTORCH_VERSION, SAGEMAKER_TRANSFORMERS_VERSION versions once we release new DLCs?

[pacman100]

LGTM! How should we maintain the SAGEMAKER_PYTHON_VERSION, SAGEMAKER_PYTORCH_VERSION, SAGEMAKER_TRANSFORMERS_VERSION versions once we release new DLCs?

Hello, as it is not a frequent change, it can be done manually for time being whenever new releases of DLC happen. Any suggestions/best practices for automating it?

[HebaGamalElDin]

Hello @pacman100, what is the sagemaker SDK estimator supposed to be used with accelerate?