aws_kinesis_firehose_delivery_stream does not manage s3_backup_mode

[mgeggie]

Terraform Version

Terraform v0.10.0

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Terraform Configuration Files

provider "aws" {
  profile = "terraform"
  region  = "us-west-2"
}

resource "aws_s3_bucket" "firehose" {
  bucket = "mgeggie-firehose"
  acl    = "private"
}

resource "aws_iam_role" "firehose" {
  name = "firehose"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "firehose" {
  name   = "firehose"
  role   = "firehose"
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
            ],
            "Resource": [
                "${aws_s3_bucket.firehose.arn}",
                "${aws_s3_bucket.firehose.arn}/*"
            ]
        },
        {
           "Effect": "Allow",
           "Action": [
               "es:DescribeElasticsearchDomain",
               "es:DescribeElasticsearchDomains",
               "es:DescribeElasticsearchDomainConfig",
               "es:ESHttpPost",
               "es:ESHttpPut"
           ],
          "Resource": [
              "${aws_elasticsearch_domain.firehose.arn}",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/*"
          ]
       },
       {
          "Effect": "Allow",
          "Action": [
              "es:ESHttpGet"
          ],
          "Resource": [
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_all/_settings",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_cluster/stats",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/index-name*/_mapping/type-name",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_nodes",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_nodes/stats",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_nodes/*/stats",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/_stats",
              "${aws_elasticsearch_domain.firehose.arn}/${aws_elasticsearch_domain.firehose.domain_name}/index-name*/_stats"
          ]
       },
       {
          "Effect": "Allow",
          "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:DescribeLogStreams",
            "logs:PutLogEvents"
          ],
          "Resource": [
              "arn:aws:logs:*"
          ]
       }
    ]
}
EOF
}

resource "aws_kinesis_firehose_delivery_stream" "firehose" {
  name        = "firehose"
  destination = "elasticsearch"
  s3_configuration {
    role_arn   = "${aws_iam_role.firehose.arn}"
    bucket_arn = "${aws_s3_bucket.firehose.arn}"
    prefix     = "/"
    cloudwatch_logging_options {
      enabled = true
      log_group_name  = "firehose"
      log_stream_name = "firehose"
    }
  }
  elasticsearch_configuration {
    domain_arn     = "${aws_elasticsearch_domain.firehose.arn}"
    index_name     = "firehose"
    role_arn       = "${aws_iam_role.firehose.arn}"
    type_name      = "object"
    s3_backup_mode = "ErrorDocumentsOnly"
  }
}

resource "aws_elasticsearch_domain" "firehose" {
  domain_name           = "firehose"
  elasticsearch_version = "5.3"
  cluster_config {
    instance_type = "m4.large.elasticsearch"
  }
  ebs_options {
    ebs_enabled = true
    volume_size = 512
  }
}

Debug Output

https://gist.github.com/mgeggie/f0d776a30ef5f375e7dc5e95abb11ceb

Panic Output

-- None --

Expected Behavior

The S3BackupMode of the firehose resource should have been changed from AllDocuments to FailedDocumentsOnly

Actual Behavior

The S3BackupMode was not changed

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. Change aws_kinesis_firehose_delivery_stream.firehose.elasticsearch_configuration.s3_backup_mode from AllDocuments to FailedDocumentsOnly
3. terraform apply
4. No change has been made to the firehose.

Important Factoids

Nothing of note.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• aws_kinesis_firehose_delivery_stream ignores log configuration #385

[cgarvis]

Your s3_backup_mode is ErrorDocumentsOnly instead of FailedDocumentsOnly

[guillaumk]

Hello,

I've the same issue with a change from "FailedDocumentsOnly" to "AllDocuments" (Terraform 0.10.8).
The apply is OK.

$ terraform state show module.firehose.aws_kinesis_firehose_delivery_stream.firehose_logs_stream
...
elasticsearch_configuration.0.s3_backup_mode               = AllDocuments
...

But in the AWS console, the "Backup mode" is still "FailedDocumentsOnly".
(And obviously, no data in S3 bucket).

[Hinidu]

I've also found that changing s3_backup_mode in terraform doesn't affect existing delievery stream on AWS. But S3 backup mode can't be changed using AWS Console too. So perhaps it can't be changed at all and this attribute should be marked as forcing the new resource on change?

[Adriks976]

I have the same issue with S3 backup mode. I agree with @Hinidu, this parameter requires a new resource on change because it can't be modify.

[bflad]

It appears this only affects elasticsearch_configuration -- we properly implement and acceptance test s3_backup_mode within extended_s3_configuration, redshift_configuration, and splunk_configuration. Once #6305 is adjusted, a change to s3_backup_mode within elasticsearch_configuration will properly force resource recreation instead of an update that cannot be completed. 👍

[bflad]

The above fix has been merged and will release with version 1.42.0 of the AWS provider, likely later today. 👍

[bflad]

This has been released in version 1.42.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!