provider/aws: Handle all kinds of CloudFormation stack failures

[radeksimko]

This fixes #5559 (specifically cb4c2a9) and attempts to fix #5487 (It is fixing an unreported bug which became clear as part of that issue).

Explanation

I'm treating this as WIP because I have asked AWS support for advice on how to catch reasons behind deleted stacks when user specifies OnFailure=DELETE reliably and I may modify this PR based on their response.

In worst case I'll most likely send another PR to introduce something like MaxRetry into resource.StateChangeConf which would allow me locally reducing this from hard-coded 10 seconds (if it's reached via exponential backoff) to 1 second (or something like that) at the risk of potentially being throttled by API limits, but reducing the risk of missing events that contain reasons for stack deletion.

In theory you could say that CloudFormation is slow enough that going from CREATE_IN_PROGRESS over DELETE_IN_PROGRESS to be gone should take more than 10 seconds, but it may not always be the case.

Test plan

make testacc TEST=./builtin/providers/aws TESTARGS='-run=AWSCloudFormation'

[radeksimko]

I have received response from AWS support - there seems to be a fairly clean (yet slightly confusing) solution.

If Stack ID (instead of stack name) is passed as a value of an argument StackName in DescribeStackEvents, it should be possible to query events of deleted stacks too.

I will update the PR soon.

[darnould]

Just encountered this issue, glad to see it's already been identified and in the process of being addressed.

Anything I can do to help drive this forward? Would be good to get this out.

[stack72]

@radeksimko is this is a place to test right now?

[radeksimko]

@stack72 This is now finally ready for review. Sorry for such a long delay.

Test plan

https://gist.github.com/radeksimko/961c544ab114b72c665392ed4b2ce786

[stack72]

Hi @radeksimko

Been through this tonight and it looks good to me :)

P.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.