[stepfunctions] Results of DescribeExecution inaccurate for differ execution statuses

[taesungh]

The response of DescribeExecution in AWS Step Functions can vary based on the status of the given execution.

The current output type DescribeExecutionOutputTypeDef (for SFN.Client.describe_execution) specifies all items as required which is partly contradictory.

DescribeExecutionOutputTypeDef = TypedDict(
    "DescribeExecutionOutputTypeDef",
    {
        "executionArn": str,
        "stateMachineArn": str,
        "name": str,
        "status": ExecutionStatusType,
        "startDate": datetime,
        "stopDate": datetime,
        "input": str,
        "inputDetails": CloudWatchEventsExecutionDataDetailsTypeDef,
        "output": str,
        "outputDetails": CloudWatchEventsExecutionDataDetailsTypeDef,
        "traceHeader": str,
        "mapRunArn": str,
        "error": str,
        "cause": str,
        "stateMachineVersionArn": str,
        "stateMachineAliasArn": str,
        "redriveCount": int,
        "redriveDate": datetime,
        "redriveStatus": ExecutionRedriveStatusType,
        "redriveStatusReason": str,
        "ResponseMetadata": ResponseMetadataTypeDef,
    },
)

Example query and potential responses

import boto3
sfn_client = boto3.client("stepfunctions")
history = sfn_client.describe_execution(executionArn="arn:...")

For status RUNNING

{
    "executionArn": "arn:...",
    "stateMachineArn": "arn:...",
    "name": "...",
    "status": "RUNNING",
    "startDate": datetime.datetime(...),
    "input": '{...}',
    "inputDetails": {"included": True},
    "redriveCount": 0,
    "redriveStatus": "NOT_REDRIVABLE",
    "redriveStatusReason": "Execution is RUNNING and cannot be redriven",
    "ResponseMetadata": { ... },
}

For status SUCCEEDED

{
    "executionArn": "arn:...",
    "stateMachineArn": "arn:...",
    "name": "...",
    "status": "SUCCEEDED",
    "startDate": datetime.datetime(...),
    "stopDate": datetime.datetime(...),
    "input": '{...}',
    "inputDetails": {"included": True},
    "output": '{...}',
    "outputDetails": {"included": True},
    "redriveCount": 0,
    "redriveStatus": "NOT_REDRIVABLE",
    "redriveStatusReason": "Execution is SUCCEEDED and cannot be redriven",
    "ResponseMetadata": { ... },
}

For status FAILED

{
    "executionArn": "arn:...",
    "stateMachineArn": "arn:...",
    "name": "...",
    "status": "FAILED",
    "startDate": datetime.datetime(...),
    "stopDate": datetime.datetime(...),
    "input": '{...}',
    "inputDetails": {"included": True},
    "traceHeader": "...",
    "error": "Error",
    "cause": "...",
    "redriveCount": 0,
    "redriveStatus": "REDRIVABLE",
    "ResponseMetadata": { ... },
}

For status TIMED_OUT

{
    "executionArn": "arn:...",
    "stateMachineArn": "arn:...",
    "name": "...",
    "status": "TIMED_OUT",
    "startDate": datetime.datetime(...),
    "stopDate": datetime.datetime(...),
    "input": '{...}',
    "inputDetails": {"included": True},
    "traceHeader": "...",
    "redriveStatus": "NOT_REDRIVABLE",
    "redriveStatusReason": "Execution's redrivable period has ended",
    "ResponseMetadata": { ... },
}

For status ABORTED

{
    "executionArn": "arn:...",
    "stateMachineArn": "arn:...",
    "name": "...",
    "status": "ABORTED",
    "startDate": datetime.datetime(...),
    "stopDate": datetime.datetime(...),
    "input": '{...}',
    "inputDetails": {"included": True},
    "traceHeader": "...",
    "redriveStatus": "REDRIVABLE",
    "ResponseMetadata": { ... },
}

And there is also status PENDING_REDRIVE for which I don't have an example.

Resolution

It would be great if this builder could produce a union of narrowed variants although it may be difficult to account for all possible scenarios. Otherwise, marking certain fields as NotRequired would suffice. Specifically, it appears stopDate, output, outputDetails, error, cause, traceHeader, and redriveStatusReason are all possibly not included in the response along with mapRunArn, redriveDate, stateMachineAliasArn, and stateMachineVersionArn.

Additional context

Referencing mypy-boto3-stepfunctions v1.35.46

[vemel]

Hello!

Thank you for the report. Unfortunately, there is no way to check totality of output shapes. However, it is possible to override a type parsed from botocore shapes and provide a union of potential output TypedDicts.

You could help me by writing all possible DescribeExecution...OutputTypeDefs here in a comment. Let me know if you are interested in adding an override for SFN.Client.describe_execution return type as a PR.

[vemel]

For now I decided to avoid massive changes, because builder does not support TypedDict -> Union substitution.
Instead, I improved logc for marking output TypedDict keys and NotRequired. So, DescribeExecutionOutputTypeDef should have correct totality in the next release.

[vemel]

Fixed in the latest release: https://github.com/youtype/mypy_boto3_builder/releases/tag/8.2.0

Please update mypy-boto3-stepfunctions and let me know if it works as expected.