AWS Kinesis Firehose Delivery repeatably fails on first run, success on second run

[rexroof]

Terraform Version

Terraform v0.11.3
+ provider.aws v1.9.0

I also tested this on terraform 0.10.8 with the same results.

Affected Resource(s)

Please list the resources as a list, for example:

• aws_kinesis_firehose_delivery_stream
• aws_iam_role_policy

This also seems related to a similar issue that was found in aws_ecr_repository and aws_ecr_repository_policy, issue #545

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"
}

data "aws_region" "current" { }
variable "stream_name" { default = "testing_example" }

resource "random_string" "rand" {
  length = 16
  special = false
}

resource "aws_kinesis_stream" "stream" {
  name             = "${var.stream_name}"
  shard_count      = 1
  retention_period = 24
}

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.stream_name}-${random_string.rand.result}-${data.aws_region.current.name}"
  acl    = "private"
}

resource "aws_iam_role" "firehose-role" {
  name               = "${var.stream_name}_firehose"
  assume_role_policy = <<EOF
{ "Version": "2012-10-17",
  "Statement": [ {
  "Action": "sts:AssumeRole",
  "Principal": { "Service": "firehose.amazonaws.com" },
  "Effect": "Allow",
  "Sid": "" } ]
}
EOF
}

resource "aws_iam_role_policy" "inline-policy" {
  name   = "${var.stream_name}_firehose_inline_policy"
  role   = "${aws_iam_role.firehose-role.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "${aws_s3_bucket.bucket.arn}",
        "${aws_s3_bucket.bucket.arn}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": "${aws_kinesis_stream.stream.arn}"
    }
  ]
}
EOF
}

resource "aws_kinesis_firehose_delivery_stream" "fh_stream" {
  name        = "${var.stream_name}_firehose"
  destination = "extended_s3"

  kinesis_source_configuration {
    kinesis_stream_arn = "${aws_kinesis_stream.stream.arn}"
    role_arn           = "${aws_iam_role.firehose-role.arn}"
  }

  extended_s3_configuration {
    role_arn        = "${aws_iam_role.firehose-role.arn}"
    bucket_arn      = "${aws_s3_bucket.bucket.arn}"
    buffer_size     = 1
    buffer_interval = 60
  }
}

Debug Output

debug output of first run: https://gist.github.com/rexroof/f6562cf49d7e308b21ddda4750a549fa
debug output of second run: https://gist.github.com/rexroof/f62fccb56da2734197252954ad7d372f

Expected Behavior

creation of kinesis firehose should complete successfully after waiting for iam policy to be applied

Actual Behavior

apply stopped, this error:

[WARN] Error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Role
arn:aws:iam::702151822554:role/testing_example_firehose is not authorized to perform: kinesis:DescribeStream on resource arn:aws:kinesis:
us-east-1:702151822554:stream/testing_example.

Steps to Reproduce

1. terraform apply
2. terraform apply

[bflad]

@rexroof thanks for reporting this and sorry you're running into trouble here. We have logic for retrying the creation with IAM eventual consistency issues, but apparently were missing this specific error message. I have submitted a PR to fix this: #3381

[bflad]

The fix for this has been merged into master and will be released in v1.10.0 of the AWS provider, likely next week. Thanks again for the report!

[CharlieC3]

@bflad This error seems to still be happening for Kinesis Firehose when trying to connect it to an existing AWS ElasticSearch cluster. However, the error is a bit different, and that's probably why this fix isn't working for it. The workaround, just like this ticket, is to run terraform apply again after the first failure, and it'll run. I'll open a new issue for it.

[bflad]

Thanks for the heads up @CharlieC3! I can get that fixed up once the new issue is in.

[CharlieC3]

@bflad Thank you! Issue is submitted here #4513

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!