Performance issue regarding Sagemaker Estimator.deploy()

[UniSabina]

System Information

• Python Version: 3.6
• CPU or GPU: CPU
• Python SDK Version: Anaconda

Describe the problem

I'm quite new to the SageMaker algorithms and estimators so please bear with me.
I'm running a script very similar to this example script for DeepAR
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_electricity/DeepAR-Electricity.ipynb
And want to start more than hundred of such training + prediction jobs.
The cell

predictor = estimator.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge',
    predictor_cls=DeepARPredictor)

takes up 70% (~8.5min) of the time of the overall training and predicting job (~12min). Is there a possibility to reduce that time? What is the reason for this deploy job taking so long?

Thanks!

[ChoiByungWook]

Hello @UniSabina,

Apologies on the slow experience.

I'll reach out to the team that handles the hosting platform and let them comment.

[vedal]

I'm also interested in this

[lexxx233]

Any update on this?

[rishabh1212]

Hii @ChoiByungWook,
Any update on this?

[nrapanos]

Is there any update on this? I'm experiencing a similar issue.

[dsow86]

Any update about this issue? I am still having the same problem, the model deployment take too long.

[zlufy90]

It is too slow for training and deploying model via SageMaker Studio. I just tested with Iris dataset.
Especially when debugging. It makes us give up easily.

[yitzchak-ben-ezra-ecoplant]

really slow

[avi-shapiro]

Still no update? I can build and push a custom PyTorch training image, and then train a model in a shorter time than deploying.

[hinthornw]

Takes over 90 minutes for me.

[Soroush-aali-bagi]

How much should I wait before killing the process?!

[justintimewatkins]

so slow still

[ArthurDeleu]

terribly slow... tring to deploy an ml.m4.xlarge, since the max health_check_timeout is topped at 3600 and my deployment currently takes over an hour, I am not able to deploy llama3 70B. What to do?