sagemaker: Support asynchronous endpoints

[petermeansrock]

Describe the feature

As described in the SageMaker Endpoint L2 construct RFC:

Asynchronous Inference: By default, a deployed endpoint is synchronous: a customer issues an InvokeEndpoint operation to SageMaker with an attached input payload and the resulting response contains the output payload from the endpoint. To instead support asynchronous invocation, the AsyncInferenceClientConfig CloudFormation attribute was added to the endpoint config resource. To interact with an asynchronous endpoint, a customer issues an InvokeEndpointAsync operation to SageMaker with an attached input location in S3; SageMaker asynchronously reads the input from S3, invokes the endpoint, and writes the output to an S3 location specified within the AsyncInferenceClientConfig attribute.

Please 👍 this issue to help with the prioritization of this feature.

Use Case

"This option is ideal for requests with large payload sizes (up to 1GB), long processing times (up to 15 minutes), and near real-time latency requirements. Asynchronous Inference enables you to save on costs by autoscaling the instance count to zero when there are no requests to process, so you only pay when your endpoint is processing requests." (link)

Proposed Solution

As described in the SageMaker Endpoint L2 construct RFC:

As discussed with the RFC bar raiser here, there are a few ways to tackle the addition of this functionlity. One option is to add attribute(s) to the L2 endpoint config construct to support asynchronous inference along with synthesis-time error handling to catch configuration conflicts (e.g., asynchronous endpoints are only capable of supporting a single instance-based production variant today). Alternatively, an AsyncEndpointConfig subclass of EndpointConfig could be introduced to provide a better compile-time contract to customers (while still implementing the generic functionality within EndpointConfig). Either way, the proposed contracts would only undergo backward-compatible changes.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.54.0-alpha.0

Environment details (OS name and version, etc.)

macOS Ventura

[perrozzi]

hello, would this example by chance help to speed up the implementation of this feature?
https://github.com/aws-solutions-library-samples/guidance-for-generative-ai-deployments-using-amazon-sagemaker-jumpstart/blob/main/simple_generative_ai_service/jumpstart_async_endpoint_construct.py