resource/aws_kinesis_firehose_delivery_stream: Retry on additional IAM eventual consistency errors

[stephencoe]

I am seeing intermittent failures when using the aws_kinesis_firehose_delivery_stream. The error reported is:

error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Failed to describe Elasticsearch domain due to: The security token included in the request is invalid.

This looks different to the current list of errors being tracked in https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_kinesis_firehose_delivery_stream.go#L2081

The message is specific about security token, the application (codebuild in this example) has permissions since this build is successful on other runs.

Possibly related issues
#3380
#4513

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Terraform Configuration Files

resource "aws_kinesis_firehose_delivery_stream" "firehose_stream" {
  name        = "${var.firehose_stream_name}"
  destination = "elasticsearch"

  s3_configuration {
    role_arn        = "${module.firehose_role.role_arn}"
    bucket_arn      = "${data.aws_s3_bucket.logs.arn}"
    buffer_size     = "${var.s3_buffer_size}"
    buffer_interval = "${var.s3_buffer_interval}"
    kms_key_arn     = "${data.aws_kms_key.s3.arn}"

    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = "${aws_cloudwatch_log_group.this.name}"
      log_stream_name = "${aws_cloudwatch_log_stream.firehose_to_s3.name}"
    }
  }

  elasticsearch_configuration {
    domain_arn         = "${var.elasticsearch_arn}"
    role_arn           = "${module.firehose_role.role_arn}"
    index_name         = "${var.firehose_stream_name}"
    type_name          = "doc"
    s3_backup_mode     = "AllDocuments"
    buffering_size     = "${var.elasticesearch_buffer_size}"
    buffering_interval = "${var.elasticesearch_buffer_interval}"

    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = "${aws_cloudwatch_log_group.this.name}"
      log_stream_name = "${aws_cloudwatch_log_stream.firehose_to_elasticsearch.name}"
    }
  }
}

Debug Output

Panic Output

Expected Behavior

Retry on error

Actual Behavior

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[lc-thirai]

Now I'm having same error.

Terraform v0.12.5
+ provider.aws v2.23.0

[pavloos]

👍 Just hit the same issue.

Note on those "possibly related issues" -they are not related at all.
Let's take #3380 for example:

kinesis_source_configuration {
    kinesis_stream_arn = "${aws_kinesis_stream.stream.arn}"
    role_arn           = "${aws_iam_role.firehose-role.arn}"
  }

The above block references ${aws_iam_role.firehose-role.arn}" however, at that point that role may not have any policy yet. The policy is created in aws_iam_role_policy.inline-policy hence it is simply dependency issue which can be resolved by constructing role arn like this:

kinesis_source_configuration {
    kinesis_stream_arn = "${aws_kinesis_stream.stream.arn}"
    role_arn                   = "${format(
        "arn:aws:iam::%s:role/%s",
        ${data.aws_caller_identity.current.account_id},
        ${aws_iam_role_policy.inline-policy.role}
    )
}

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.