AWS Kinesis Firehose Delivery repeatably fails on first run when connecting to AWS ElasticSearch, success on second run

[CharlieC3]

This error is very similar to #3380, however the error I'm getting is slightly different because Kinesis Firehose is trying to connect to ElasticSearch, unlike the original ticket. The workaround, just like the original ticket, is to run terraform apply again after the first failure, and it'll run successfully.

Terraform Version

Terraform v0.11.7
provider.aws v1.18.0

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream
• aws_iam_role_policy
• aws_elasticsearch_domain

Terraform Configuration Files

https://gist.github.com/CharlieC3/4a211d185000285f206d890044889e73

Debug Output

https://gist.github.com/CharlieC3/703b219cf03231d6ae7aeecfaff49290

Expected Behavior

Terraform should have retried the creation of the Kinesis Firehose Delivery Stream due to the fix created from issue #3380.

Actual Behavior

Terraform did not retry the creation of the Kinesis Firehose Delivery Stream. This is purely a guess without doing much digging in the Terraform source code, but this error is possibly happening because the actual error message I received was worded differently than what the fix is looking for.
The error I received says:

InvalidArgumentException: Delivery Role is not authorized to describe the Elasticsearch domain

And the fix linked below is worded to look for an error which says:

is not authorized to perform

https://github.com/terraform-providers/terraform-provider-aws/blob/v1.18.0/aws/resource_aws_kinesis_firehose_delivery_stream.go#L1429

Note the error I received says "describe" and the error the fix looks for says "perform".

Steps to Reproduce

1. terraform apply

References

• Original Issue AWS Kinesis Firehose Delivery repeatably fails on first run, success on second run #3380
• Original PR resource/aws_kinesis_firehose_delivery_stream: Retry on additional IAM eventual consistency errors #3381

[bflad]

Pull request submitted: #4518 -- thanks for submitting this issue!

[bflad]

The fix for this is now in and will release later this week in v1.19.0 of the AWS provider.

[bflad]

This has been released in version 1.19.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[CharlieC3]

@bflad Already used the fixed earlier today :) thank you again for the quick turn-around!

[ahl]

@bflad I'm seeing a similar problem

error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Verify that the IAM role has access to the ElasticSearch domain

Perhaps AWS has changed its error message or this is another related pathology?

[ahl]

@bflad ping ^^

[alejandrox1]

Getting the same error as @ahl when trying the elasticsearch destination example from https://www.terraform.io/docs/providers/aws/r/kinesis_firehose_delivery_stream.html
any chance these are related?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!