[datadog_integration_aws_*] Validate AWS account ID and improve error handling

[rjhornsby]

Attempting to resolve issue #2200

Until sometime recently, fedramp DD API required the AWS account ID argument to be an IAM access key ID when the TF provider resource asked for the AWS account_id. This seems to have changed to match the commercial behavior where the account id argument is expected to get the actual account ID.

However, the API change broke the provider in a way that the provider cannot handle. It ends up crashing in the middle of the terraform apply:

Error: Provider produced inconsistent result after apply
│
│ When applying changes to datadog_integration_aws_lambda_arn.datadog-forwarder-global, provider "provider[\"registry.terraform.io/datadog/datadog\"]" produced an unexpected new value: Root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

This PR does not address the underlying handling of the invalid/unhandled API response, but instead tries to avoid it by ensuring the AWS account ID is a 12 digit string, and providing useful feedback if it does not - ie in the case of code which might still be using the IAM access key.

[nkzou]

Hi there, thanks for the PR and raising the issue on the breaking change.
We're currently verifying some things internally with the aws integration API, and I'm also working on a small refactor + better error handling for aws resources so we can avoid the unhelpful "inconsistent state" errors. Should have an update soon.

[andrew-purdin]

@nkzou is everything fixed with this now, or are there more API changes coming? We also had a sudden break in our Datadog/AWS integration IaC from the related issue here.

[nkzou]

Things should be stable now. Are you running into issues still?

[andrew-purdin]

Yes, but we are stuck on version 3.30.0 of the provider for the time being and are unable to upgrade. I expect we are continuing to see issues due to the recent API changes not being backward compatible with 3.30.0, is that correct?

[nkzou]

Yes, unfortunately #2154 is the PR with the key fix for this issue (the PR we're on is just validation/error handling). The earliest version with the bugfix is 3.32.0.

[andrew-purdin]

Thanks for clarifying. Just out of curiosity, is there a source we can follow for breaking API changes like this on the DD side? It was tough for us to determine the source of this problem at first.

[rjhornsby]

Thanks for clarifying. Just out of curiosity, is there a source we can follow for breaking API changes like this on the DD side? It was tough for us to determine the source of this problem at first.

+1. I spent a few hours tracking this down by diving into the provider code and then the API's behavior to finally figure out it was an API change. Because we couldn't figure out what was going on, DD support also had to spend time internally running this down.

At minimum the folks responsible for the API need to please communicate with the internal consumers like the folks responsible for the TF provider that an API change is coming so this doesn't send folks like @andrew-purdin and myself running in circles trying to understand what we did wrong.

[nkzou]

Breaking changes like this should only happen in very specific and rare cases, so there isn't a specific place to look for updates about them. With that being said, we've received your feedback about this and have passed it along to the relevant team.